Systems and methods for hyperspectral imaging

ABSTRACT

An apparatus for analyzing a subject including a hyperspectral image module is provided. It is used to identify a suspect region of a subject by using a hyperspectral sensor (for obtaining a hyperspectral image of the subject), a control computer including a processor unit (PU) and a computer readable memory (CRM) (for controlling and is in electronic communication with the sensor), a control software module including instructions stored in the CRM and executed by the PU (for controlling said at least one operating parameter of the sensor), a spectral calibrator module including instructions stored in the CRM and executed by the PU (for applying a wavelength dependent spectral calibration standard constructed for the sensor to a hyperspectral image), and a light source for illuminating the subject. An optional contact probe module is used to collect a signal of the suspect region for medical diagnosis.

This application is a continuation of U.S. patent application Ser. No.12/370,557, filed Feb. 12, 2009, which claims priority to U.S.Provisional Patent Application No. 61/073,527, filed Jun. 18, 2008, andentitled “Systems and Methods for Hyperspectral Medical Imaging,” theentire contents of which are hereby incorporated by reference herein.

FIELD OF THE APPLICATION

This application generally relates to systems and methods for medicalimaging.

BACKGROUND

Affecting more than one million Americans each year, skin cancer is themost prevalent form of cancer, accounting for nearly half of all newcancers reported, and the number is rising. However, according to theAmerican Academy of Dermatology, most forms of skin cancer are almostalways curable when found and treated early. For further details, see A.C. Geller et al., “The first 15 years of the American Academy ofDermatology skin cancer screening programs: 1985-1999,” Journal of theAmerican Academy of Dermatology 48(1), 34-41 (2003), the entire contentsof which are hereby incorporated by reference herein. As the number ofsubjects diagnosed with skin cancer continues to rise year-by-year,early detection and delineation are increasingly useful.

During a conventional examination, dermatologists visually survey theskin for lesions or moles that fit certain pre-defined criteria for apotential malignant condition. If an area is suspect, the doctor willperform a biopsy, sending the tissue to a pathology lab for diagnosis.Though effective, this method of detection is time consuming, invasive,and does not provide an immediate definitive diagnosis of a suspectlesion. It is also vulnerable to false positives which introduceunnecessary biopsy and associated costs. More importantly, earlydetection is very difficult at best, as developing cancers are notusually visible without close inspection of the skin.

Medical imaging has the potential to assist in the detection andcharacterization of skin cancers, as well as a wide variety of otherconditions.

Hyperspectral medical imaging is useful because, among other things, itallows information about a subject to be obtained that is not readilyvisible to the naked eye. For example, the presence of a lesion may bevisually identifiable, but the lesion's actual extent or what type ofcondition it represents may not be discernable upon visual inspection,or for that matter whether the lesion is benign or cancerous. Althoughtentative conclusions about the lesion can be drawn based on somegeneral visual indicators such as color and shape, generally a biopsy isneeded to conclusively identify the type of lesion. Such a biopsy isinvasive, painful, and possibly unnecessary in cases where the lesionturns out to be benign.

In contrast, hyperspectral medical imaging is a powerful tool thatsignificantly extends the ability to identify and characterize medicalconditions. “Hyperspectral medical imaging” means utilizing multiplespectral regions to image a subject, e.g., the entire body or a bodypart of a human or animal, and thus to obtain medical information aboutthat subject. Specifically, each particular region of a subject has aunique spectral signature extending across multiple bands of theelectromagnetic spectrum. This spectral signature contains medical,physiological, and compositional information about the correspondingregion of the subject. For example, if the subject has a cancerous skinlesion, that lesion may have a different color, density, and/orcomposition than the subject's normal skin, thus resulting in the lesionhaving a different spectrum than the normal skin. While thesedifferences may be difficult to visually detect with the naked eye, thedifferences may become apparent through spectroscopic analysis, thusallowing the lesion (or other medical condition resulting in ameasurable spectroscopic feature) to be identified, characterized, andultimately more readily treated than would be possible usingconventional visual inspection and biopsy. Such spectral differences canbe presented to a user (such as a physician), for example, byconstructing a two-dimensional image of the lesion. See, for example,U.S. Pat. No. 6,937,885, the entire contents of which are herebyincorporated by reference.

However, the potential applicability of conventional systems and methodsfor hyperspectral medical imaging has been limited by the types ofsensors and analytical techniques used. What are needed are morepowerful and robust systems and methods for collecting, analyzing, andusing hyperspectral information to diagnose and treat subjects.

SUMMARY

Embodiments of the application provide systems and methods of spectralmedical imaging. Alternative embodiments of the application also providesystems and methods of analysis by a contact probe based on a resultfrom the prior spectral medical imagine analysis.

One aspect provides a method of hyperspectral medical imaging fordiagnosing a medical condition of a subject. One embodiment provides anapparatus for analyzing the skin of a subject comprising (i) ahyperspectral sensor for obtaining a hyperspectral image of the subject,(ii) a control computer for controlling the hyperspectral sensor, wherethe control computer is in electronic communication with thehyperspectral sensor and where the control computer controls at leastone operating parameter of the hyperspectral sensor, and where thecontrol computer comprises a processor unit and a computer readablememory, (iii) a control software module, stored in the computer readablememory and executed by the processor unit, the control softwarecomprising instructions for controlling the at least one operatingparameter of the hyperspectral sensor, (iv) a spectral calibratormodule, stored in the computer readable memory and executed by theprocessor unit, the spectral calibrator module comprising instructionsfor applying a wave-length dependent spectral calibration standardconstructed for the hyperspectral sensor to a hyperspectral imagecollected by the hyperspectral sensor, and (v) a light source thatilluminates the skin of the subject for the hyperspectral sensor.

Under another aspect, an apparatus for analyzing the skin of a subjectincludes: a hyperspectral sensor for obtaining a hyperspectral image ofthe subject; a control computer for controlling the hyperspectralsensor, where the control computer is in electronic communication withthe hyperspectral sensor and where the control computer controls atleast one operating parameter of the hyperspectral sensor, and where thecontrol computer includes a processor unit and a computer readablememory; a control software module, stored in the computer readablememory and executed by the processor unit, the control softwareincluding instructions for controlling the at least one operatingparameter of the hyperspectral sensor; a spectral calibrator module,stored in the computer readable memory and executed by the processorunit, the spectral calibrator module including instructions for applyinga wavelength dependent spectral calibration standard constructed for thehyperspectral sensor to a hyperspectral image collected by thehyperspectral sensor; and a light source that illuminates the skin ofthe subject for the hyperspectral sensor.

In some embodiments, the at least one operating parameter is a sensorcontrol, an exposure setting, a frame rate, or an integration rate. Insome embodiments, a power to the light source is controlled by thecontrol software module. In some embodiments, the apparatus furtherincludes one or more batteries for powering the hyperspectral sensor,the control computer and the light source, where the apparatus isportable. In some embodiments, the apparatus further includes a scanmirror to provide simulated motion for a hyperspectral scan of the skinof the subject. In some embodiments, the light source includes apolarizer. In some embodiments, the hyperspectral sensor includes across polarizer. In some embodiments, the hyperspectral includes asensor head, and the control software module includes instructions formoving the sensor head through a range of distances relative to thesubject, including a first distance that permits a wide field view of aportion of the subject's skin, and a second distance that permits adetailed view of a portion of the subject's skin. In some embodiments,the hyperspectral sensor is mounted on a tripod. In some embodiments,the tripod is a fixed sensor tripod or a fixed sensor tripod on wheels.In some embodiments, the hyperspectral sensor is mounted on a mobilerack. In some embodiments, the hyperspectral sensor can be mounted onone or more of a scanning bed and an articulated arm. Alternatively, thehyperspectral sensor can be handheld and include fiber optics that aremanually moved to obtain reflected light from different portions of thesubject.

In some embodiments, the apparatus further includes: a plurality ofsignatures, each signature in the plurality of signatures correspondingto a characterized human lesion; and a spectral analyzer module storedin the computer readable memory, the spectral analyzer module includinginstructions for comparing a spectrum acquired using the hyperspectralsensor to a signature in the plurality of signatures. In someembodiments, the apparatus further includes a trained data analysisalgorithm, stored in the computer readable memory, for identifying aregion of the subject's skin of biological interest using an imageobtained by the apparatus. In some embodiments, the trained dataanalysis algorithm is a trained neural network, a trained support vectormachine, a decision tree, or a multiple additive regression tree. Insome embodiments, the apparatus further includes a trained data analysisalgorithm, stored in the computer readable memory, for characterizing aregion of the subject's skin of biological interest using an imageobtained by the apparatus. In some embodiments, the trained dataanalysis algorithm is a trained neural network, a trained support vectormachine, a decision tree, or a multiple additive regression tree. Insome embodiments, the apparatus further includes a trained data analysisalgorithm, stored in the computer readable memory, for determining aportion of a hyperspectral data cube that contains information about abiological insult in the subject's skin. In some embodiments, thetrained data analysis algorithm is a trained neural network, a trainedsupport vector machine, a decision tree, or a multiple additiveregression tree.

In some embodiments, the apparatus further includes: a storage module,stored in the computer readable media, where the storage module includesa plurality of spectra of the subject's skin taken at different timepoints; and an analysis module, stored in the computer readable media,where the analysis module includes instructions for using the pluralityof spectra to form a normalization baseline of the skin. In someembodiments, the different time points span one or more contiguousyears. In some embodiments, the analysis module further includesinstructions for analyzing the plurality of spectra to determine a timewhen a biological insult originated. In some embodiments, the biologicalinsult is a lesion.

In some embodiments, the apparatus further includes a sensor other thana hyperspectral sensor. In some embodiments, the other sensor is adigital camera, a LIDAR sensor, or a terahertz sensor. In someembodiments, the apparatus further includes a fusion module, stored inthe computer readable memory, for fusing an image of a portion of theskin of the subject from the other sensor and an image of a portion ofthe skin of the subject from the hyperspectral sensor. In someembodiments, the fusion module includes instructions for color coding orgreyscaling data from the image of a portion of the skin of the subjectfrom the hyperspectral sensor onto the image of a portion of the skin ofthe subject from the other sensor. In some embodiments, the fusionmodule includes instructions for color coding or greyscaling data fromthe image of a portion of the skin of the subject from the other sensoronto the image of a portion of the skin of the subject from thehyperspectral sensor. In some embodiments, the fusion module includesinstructions for color coding or greyscaling data from the image of aportion of the skin of the subject from the other sensor as well ascolor coding or greyscaling data from the image of a portion of the skinof the subject from the hyperspectral sensor.

Some embodiments further include an integrated display for displayingdata from the hyperspectral sensor and a value of the at least oneoperating parameter that is controlled by the control computer. In someembodiments, the integrated display further displays the probabilisticpresence of a biological insult to the skin of the subject.

Some embodiments further include a spectral analyzer module, stored inthe computer readable media, where the spectral analyzer module includesinstructions for determining a boundary of an image of a biologicalinsult in the hyperspectral image. In some embodiments, the boundary ofthe image of the biological insult is manually determined by a user. Insome embodiments, the boundary of the image of the biological insult isdetermined by a trained data analysis algorithm. Some embodimentsfurther segments the image of the biological insult into multiplesegments. Some embodiments further include a communications module, thecommunications module including instructions for communicating theboundary of the image of the biological insult to a local or remotecomputer over a network connection. In some embodiments, thecommunications module further includes instructions for communicating aframe of reference of the skin of the subject with the boundary of theimage of the biological insult to the local or remote computer over thenetwork connection.

Under another aspect, a method of diagnosing a medical condition in asubject, the subject having a plurality of regions, includes: obtaininglight from each region of the plurality of regions without regard to anyvisible characteristics of the plurality of regions; resolving the lightobtained from each region of the plurality of regions into acorresponding spectrum; based on a stored spectral signaturecorresponding to the medical condition, obtaining a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; if the probability exceeds a pre-definedthreshold, displaying an indicator representing the probable presence ofthe medical condition in the corresponding region.

Under another aspect, a method of diagnosing a medical condition insubject, the subject having a plurality of regions, includes: resolvinglight obtained from each region of the plurality of regions into acorresponding spectrum; based on a stored spectral signaturecorresponding to the medical condition, obtaining a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; if the probability exceeds a first pre-definedthreshold, displaying an indicator representing the probable presence ofthe medical condition in the corresponding region; accepting user inputsetting a second pre-defined threshold; and if the probability exceedsthe second pre-defined threshold, displaying an indicator representingthe probable presence of the medical condition in the correspondingregion.

Under another aspect, a method of diagnosing a medical condition insubject, the subject having a plurality of regions, includes: resolvinglight obtained from each region of the plurality of regions into acorresponding spectrum; based on a stored spectral signaturecorresponding to the medical condition, obtaining a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; if the probability exceeds a first pre-definedthreshold, displaying an indicator representing the probable presence ofthe medical condition in the corresponding region, and displaying atleast one of a type of the medical condition, a category of the medicalcondition, an age of the medical condition, a boundary of the medicalcondition, and a new area of interest for examination.

Under another aspect, a method of diagnosing a medical condition in asubject includes: at a first distance from the subject, obtaining lightfrom each region of a first plurality of regions of the subject;resolving the light obtained from each region of the first plurality ofregions into a corresponding spectrum; based on a spectralcharacteristic present in a subset of the first plurality of regions,determining a second distance from the subject allowing for closerexamination of the subset; at a second distance from the subject,obtaining light from each region of a second plurality of regions of thesubject, the second plurality of regions including the subset; resolvingthe light obtained from each region of the second plurality of regionsinto a corresponding spectrum; based on a stored spectral signaturecorresponding to the medical condition, obtaining a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; and if the probability exceeds a pre-definedthreshold, displaying an indicator representing the probable presence ofthe medical condition in the corresponding region.

Under another aspect, a method of characterizing a medical condition ina subject, the subject having a plurality of regions, includes: at afirst time, resolving light obtained from each region of the pluralityof regions into a corresponding spectrum; storing the spectracorresponding to the first time; at a second time subsequent to thefirst time, resolving light obtained from each region of the pluralityof regions into a corresponding spectrum; based on a comparison of thespectra corresponding to the second time to the spectra corresponding tothe first time, determining that the medical condition had been presentat the first time although it had not been apparent at the first time;and displaying an indicator representing the probable presence of themedical condition in the subject.

One alternative embodiment provides an apparatus for analyzing the skinof a subject comprising (i) a hyperspectral sensor for obtaining ahyperspectral image of the subject, (ii) a control computer forcontrolling the hyperspectral sensor, where the control computer is inelectronic communication with the hyperspectral sensor and where thecontrol computer controls at least one operating parameter of thehyperspectral sensor, and where the control computer comprises aprocessor unit and a computer readable memory, (iii) a control softwaremodule, stored in the computer readable memory and executed by theprocessor unit, the control software comprising instructions forcontrolling the at least one operating parameter of the hyperspectralsensor, (iv) a spectral calibrator module, stored in the computerreadable memory and executed by the processor unit, the spectralcalibrator module comprising instructions for applying a wave-lengthdependent spectral calibration standard constructed for thehyperspectral sensor to a hyperspectral image collected by thehyperspectral sensor, (v) a light source that illuminates the skin ofthe subject for the hyperspectral sensor, and (vi) a contact probemodule in electronic communication with the control computer, where thecontact probe module is configured to measure a region of interest ofthe skin of the subject that is identified based on an aspect of thehyperspectral image, and where the contact probe module comprises afirst probe head unit connected to a handle unit.

In some alternative embodiments, the contact probe module furthercomprises a connector unit connected to the handle unit. In somealternative embodiments, the first probe head unit is configured toobtain a signal from a region of interest of the skin of the subject,where the signal is an electrical, a magnetic, a thermal, a mechanical,a radiometric, a photometric, or an optical signal. In some alternativeembodiments, the handle unit is configured to receive the first probehead unit and a second probe head unit, where the first probe head unitis a different type of probe head than the second probe head unit. Insome alternative embodiments, the first probe head unit is an electricalprobe, a magnetic probe, a thermal probe, a mechanical probe, aradiometric probe, a photometric probe or an optical probe. In somealternative embodiments, the contact probe module is configured toobtain more than one type of signal from the region of interest of theskin of the subject. In some alternative embodiments, the contact probemodule is configured to measure at least a first type of signal and asecond type of signal, where the first type of signal is different thanthe second type of signal. In some alternative embodiments, the contactprobe module is configured to measure the region of interest of the skinof the subject at a plurality of different times.

One alternative aspect provides a method of diagnosing a medicalcondition of a subject using a contact probe module, the subject havinga plurality of regions of skin of interest, the method comprising: (A)obtaining light from each region of skin in the plurality of regions ofskin of interest; (B) resolving light obtained in the obtaining (A) fromeach region of skin in the plurality of regions of skin into acorresponding spectrum using a hyperspectral module, therebyconstructing a plurality of spectra; (C) identifying a suspect region ofskin in the plurality of regions of skin by obtaining a first indicationthat a first spectrum in the plurality of spectra that corresponds tothe suspect region includes an indicia of the medical condition beingpresent in the suspect region; (D) collecting, after the identifying(C), a measurement of the suspect region using the contact probe modulethereby obtaining a second indication that the medical condition ispresent in the suspect region, where the contact probe module isconfigured to obtain a signal from the suspect region; and (E)outputting an indicator representing the presence of the medicalcondition in the suspect region to a user, a user interface device, or amonitor; or outputting an indicator representing the presence of themedical condition in the suspect region in user readable form to acomputer readable storage medium or a local or remote computer system;or displaying an indicator representing the presence of the medicalcondition in the suspect region. In some embodiments, the light has beenpreviously obtained and stored in the alternative hyperspectralapparatus.

In some alternative embodiments, the first indication is made based on acomparison between the first spectrum and a stored predeterminedspectral signature corresponding to the medical condition and where thestored predetermined spectral signature is obtained from a region thatis known to have the medical condition. In some alternative embodiments,the first indication is made based on a comparison between the firstspectrum and a stored predetermined spectral signature corresponding tothe medical condition and where the first indication is made when thefirst spectrum shares a similarity to the stored predetermined spectralsignature. In some alternative embodiments, the similarity is determinedbased on a similarity metric computed between the first spectrum and thestored predetermined spectral signature. In some alternativeembodiments, the first indication is made based on a comparison betweenthe first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition and where the second indicationis made when the signal from the suspect region shares a similarity tothe stored measurement signature. In some alternative embodiments, thesimilarity is determined based on a similarity metric computed betweenthe signal from the suspect region and the stored measurement signature.In some embodiments, the first indication is made based on a comparisonbetween the first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition. In some alternative embodiments,the second indication is made based on comparison between themeasurement and a stored measurement signature corresponding to themedical condition. In alternative some embodiments, the signal is anelectrical signal, a magnetic signal, a thermal signal, a mechanicalsignal, or an optical signal.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a method for diagnosing a subject using spectralmedical imaging, according to some embodiments. Steps in the dashedrectangle depict alternative embodiments that include use of a contactprobe.

FIG. 1B illustrates a method for obtaining a spectral image of asubject, according to some embodiments.

FIG. 2A schematically illustrates a system for spectral medical imaging,according to some embodiments.

FIG. 2B schematically illustrates components of a system for spectralmedical imaging, according to some embodiments.

FIG. 2C depicts an exemplary contact probe module, comprising a headunit, a handle unit, and a connector unit, according to someembodiments.

FIG. 3A schematically illustrates a hyperspectral data “plane” includingmedical information about a subject, according to some embodiments.

FIG. 3B schematically illustrates a hyperspectral data “cube” includingmedical information about a subject, according to some embodiments.

FIG. 4A schematically illustrates selection of a portion of ahyperspectral data “cube” including medical information about a subject,according to some embodiments.

FIG. 4B schematically illustrates a selected portion of a hyperspectraldata “cube” including medical information about a subject, according tosome embodiments.

FIG. 5 schematically illustrates an image based on a portion of aspectrum, according to some embodiments.

FIG. 6 schematically illustrates an embodiment of a processingsubsystem, according to some embodiments.

FIGS. 7A-7C illustrate exemplary images from different spectral bandsthat contain different medical information about a subject, according tosome embodiments.

FIG. 8A illustrates a method of using a personalized database ofspectral information for a subject, according to some embodiments.

FIG. 8B illustrates an exemplary database of spectral information forone or more subjects, according to some embodiments.

FIG. 9 illustrates a method of obtaining temporal information about acondition, according to some embodiments.

FIG. 10 illustrates a method of using pattern classification techniques,according to some embodiments.

DETAILED DESCRIPTION

Embodiments of the application provide systems and methods for spectralmedical imaging.

Specifically, the present application provides systems and methods thatenable the diagnosis of a medical condition in a subject using spectralmedical imaging data obtained using any combination of sensor such as aLIDAR sensor, a thermal imaging sensor, a millimeter-wave (microwave)sensor, a color sensor, an X-ray sensor, a UV sensor, a NIR sensor, aSWIR sensor, a MWIR sensor, a LWIR sensor, and/or a hyperspectral imagesensor. For example, a hyperspectral image of the subject can beobtained by irradiating a region of the subject with a light source, andcollecting and spectrally analyzing the light from the subject. An imagethat maps the spectrally analyzed light onto visible cues, such as falsecolors and/or intensity distributions, each representing spectralfeatures that include medical information about the subject is thengenerated based on the spectral analysis. Those visible cues, thehyperspectral image, can be displayed in “real time” (that is,preferably with an imperceptible delay between irradiation and display),allowing for the concurrent or contemporaneous inspection of both thesubject and the spectral information about the subject. From this, adiagnosis can be made and a treatment plan can be developed for thesubject.

Optionally, the spectral image includes not only the visible cuesrepresenting spectral information about the subject, but also othertypes of information about the subject. For example, a conventionalvisible-light image of the subject can be obtained, and the spectralinformation overlaid on that conventional image in order to aid incorrelation between the spectral features and the regions that generatedthose features. Or, for example, information can be obtained frommultiple types of sensors (e.g., LIDAR, color, thermal, THz) and thatinformation combined with the hyperspectral image, thus concurrentlyproviding different, and potentially complementary types of informationabout the subject. Based on information in the hyperspectral imageand/or from other types of sensors, one or more sensors or analyticalparameters can be modified and new images obtained, in order to moreaccurately make a diagnosis.

First, an overview of methods of making a medical diagnosis will beprovided. Then, a system for spectral medical imaging will be describedin detail. Then, various potential applications of spectral medicalimaging will be described. Lastly, some examples of other embodimentswill be described. The described methods, systems, applications, andembodiments are intended to be merely exemplary, and not limiting.

1. Overview of Methods

FIG. 1A illustrates an overview of a method 100 of making a medicaldiagnosis using medical imaging. First, a subject is examined (101). Theexamination can include visually observing, smelling, and/or touchingthe subject, as is conventionally done in medical examinations. Aparticular area of the subject's skin may be focused on, based on thesubject's complaints and/or based on observations made of the subject.

Then, a spectral image of the subject (102) is taken, for example, animage of a particular area of the subject's skin of interest. In someembodiments, the particular area of the subject comprises severaldifferent contiguous or non-contiguous regions of the subject's skin. Asdescribed in greater detail below, in some embodiments, this image is ahyperspectral image that is obtained by irradiating the subject withlight, collecting and analyzing light from the subject, and constructinga processed hyperspectral image based on the results of the analysis.Optionally, obtaining a hyperspectral image also includes obtainingother types of information about the subject, such as images in specificspectral bands (e.g., a THz image), and fusing or otherwise combiningthat information with the hyperspectral image.

The processed image(s) are reviewed (103), for example, to determinewhether the image(s) contain any information indicating that the subjecthas a medical condition. Based on the results of the review, a suspectregion is identified among the plurality of regions as a region with anindication that the data collected at the particular region include anindicia of a medical condition (e.g., an indicia for skin melanoma maybe a particular type of a skin lesion and/or coloration that maycorrespond to the type of skin melanoma). Measurements from theplurality of regions may be processed according to methods such asclustering analysis, principal component analysis or any analyticmethods that are suitable for processing such measurements. In someembodiments, the measurements are normalized before any analyticalprocessing is performed.

In some embodiments, the indication for the suspect region is obtainedby comparing measurements from one or more of the plurality of regionswith one or more spectral signatures in a library of spectralsignatures, in which each spectral signature in the library correspondsto one or more spectral characteristics of a medical condition. In someembodiments, the suspect region is identified when the comparisonresults in a region that most resembles a spectral signature from thespectral signature library. Results from such a comparison can berepresented by a similarity metric, such as a ratio or a similaritymatrix. Examples of similarity metrics are reviewed in McGill et al.,1979, “An evaluation of factors affecting document ranking byinformation retrieval systems,” Project report, Syracuse UniversitySchool of Information Studies, which is hereby incorporated herein byreference. In particular, Table 2 of McGill et at list 67 differentexemplary similarity metrics that can be used to compare measurementsfrom one or more of the plurality of regions with a spectral signaturein a library of spectral signatures. If the similarity metrics is abovea certain value, the corresponding region will be identified as thesuspect region. For example, a region can be identified as a suspectregion when the similarity metric between the region and a spectralsignature for a medical condition is, e.g., a ratio of 0.4 or above, 0.5or above, 0.6 or above, 0.7 or above, 0.8 or above, 0.9 or above, where1.0 represents complete identity between the region and the spectralsignature of the medical condition. It will be understood by one ofskill of the art that the threshold for the similarity metric can differwith respect to the type of tissues under analysis. For example, it maybe more difficult obtaining signals for a certain type of tissues suchthat the signal to noise ratio for the particular tissue may be low;thus ultimately resulting in a low similarity metrics value. In someembodiments, measurements of the plurality of regions are resolved intospectra where each spectrum corresponds to a region. The spectrumcorresponding to a region is then compared to a signature spectrumcorresponding to a medical condition. The signature spectrum is obtainedby taking measurements of a region that is known to have the medicalcondition. It will be understood by one of skill in the art that anysuitable analytical methods can be used for obtaining the spectralsignature of the medical condition.

To compute a similarity metric (e.g., a correlation coefficient) betweenmeasurements from the plurality of regions with one or more spectralsignatures in a library of spectral signatures, any number of techniquescan be used. For example, a plurality of features can be calculated fromthe measurements of a region in the plurality of regions. These samefeatures can be calculated from the spectral signatures. One example ofthe feature is an observed or recorded intensity at a particularwavelength. Another example of a feature is the ratio of one observed orrecorded intensity at one particular wavelength divided by anotherobserved or recorded intensity at another particular wavelength. Anotherexample of a feature is some mathematical function of two or moreobserved or recorded intensities at unique wavelengths. Exemplarymathematical functions include, but are not limited to monomial andpolynomial functions (e.g., binomial functions, quadratic functions,trinomial function, etc.) of any degree greater than zero (e.g., thelinear function f(x)=a*(x)+b given above where the value └a*(x)┘ istaken for a*(x)), rational functions

$\left( {{e.g.},{{R(x)} = \frac{{a_{n}x^{n}} + {a_{n - 1}x^{n - 1}} + \ldots + {a_{1}x} + a_{o}}{{b_{n}x^{n}} + {b_{n - 1}x^{n - 1}} + \ldots + {b_{1}x} + b_{o}}}} \right),$exponential functions (e.g., exponential decay or exponential growth),power functions (e.g., f(x)=ax^(p), where a and p are real numbers),power series (e.g., a power series on variable x), or any combinationthereof. In this way, five or more, ten or more, twenty or more, or onehundred or more features can be calculated from each hyperspectralimage. The values of these features can then be used as the basis forcomputing a similarity between such images.

In some embodiments, more than one spectral signature may be identifiedthat corresponds to one medical condition. Also, more than one spectralsignature may be identified that corresponds to more than one medicalcondition. In some embodiments, adjustments are made to the measurementsand/or analytical parameters (106) that control the imaging in order toobtain (102) one or more new or modified spectral images of the subjectwith improved quality. The new or modified spectral images of thesubject may be used to identify a suspect region for analysis by analternative hyperspectral apparatus which comprises a contact probemodule (107-109). It will be understood that a contact probe module isan apparatus to be used in conjunction with a hyperspectral moduledisclosed herein and that any methods of data collection and dataanalysis that apply the hyperspectral apparatus would also apply to thehyperspectral apparatus with a contact probe module. In someembodiments, more than one suspect region is identified and subjected tosubsequent analyses. As shown in FIG. 1A, it will be understood that anexemplary embodiment as outlined in steps 107-109 is additional andoptional to the main hyperspectral imaging process.

In some alternative embodiments, the suspect region is identified in aprevious spectral analysis such that only measurements of the suspectregion will be taken using the contact probes disclosed herein.

In alternative embodiments, after a suspect region is identified, acontact probe module is used to obtain additional data of the suspectregion (107). In some alternative embodiments, the contact probeprovides the same type of characterization of the suspect region as theoriginal hyperspectral image analysis, but at a higher resolution orwith improved signal quality. In some alternative embodiments, thecontact probe provides one or more different types of characterizationsof the suspect region such that an additional different type of datameasurement is obtained. The additional data is processed at step 108.In some embodiments, information obtained from the additional data isused to modify the previously obtained hyperspectral images. In somealternative embodiments, information obtained from the additional datais used to augment the previously obtained hyperspectral images. It isunderstood that additional procedures may be applied to the suspectregion prior to collecting the additional data. For example, the suspectregion may be cleaned (e.g., using rubbing alcohol) or processed (e.g.,using a specific dye for detecting the presence of specific biologicalmarkers).

In some alternative embodiments, additional steps may be taken withrespect to the patient or subject prior to analysis by contact probes.For example, the patient or subject may be subject to ultraviolet lightor infrared light, or an agent may be injected to the suspect site thatwill facilitate the contact probe analysis.

In some alternative embodiments, the data obtained by contact probes at107 is used to create a separate image. In some alternative embodiments,the separate image is then fused to the previously obtainedhyperspectral image. In some embodiments, the separate image is used toaugment the information obtained from the original hyperspectral image.

It will be understood that measurements obtained of the suspect regionare compared with a known measurement signature which corresponds to amedical condition. Advantageously, the measurements obtained of thesuspect region are processed before any comparison is made. As describedhereinabove, results from such comparison can be represented by asimilarity metrics. If the similarity metrics is above a certain value,the suspect region will be identified as having a certain medicalcondition. For example, a suspect region will be determined as having amedical condition when the similarity metric between the suspect regionand a measurement signature for a medical condition is, e.g., a ratio of0.4 or above, 0.5 or above, 0.6 or above, 0.7 or above, 0.8 or above,0.9 or above, where 1.0 represents complete identity betweenmeasurements of the suspect region and the measurement signature of themedical condition. It will be understood by one of skill of the art thatthe threshold for the similarity metric can differ with respect to thetype of tissues under analysis. For example, it may be more difficultobtaining signals for a certain type of tissues such that the signal tonoise ratio for the particular tissue may be low; thus ultimatelyresulting in a low similarity metric value.

In some embodiments, a medical condition can be a skin lesion that isassociated with a skin cancer, a tumor, or an internal injury. Anindicator for a medical condition can be any biological or chemicalmarker the presence or change of which suggests the presence of adisease. For example, signals corresponding to changes of hemeconcentrations in the blood may be used to diagnose whether a patienthas malaria or other diseases that involved changes of hemeconcentrations in blood. An indicia for a medical condition can be, forexample, the concentration of biological or chemical marker has exceededa threshold value, suggesting that a medical condition is present.

In the case where the image is a fusion of a hyperspectral image withanother spectral source and the image indicates the presence of amedical condition, a parameter of the hyperspectral imaging process canbe altered in order to attempt to observe the medical condition, e.g.,by seeing what spectral features are present at wavelengths other thanthose originally measured, or by seeing the area or a subset of the areawith different spatial and/or spectral resolutions.

In some embodiments, repeat measurements may be taken at thehyperspectral stage (102) and the data are averaged to reduceexperimental errors. In alternative embodiments, repeat measurements maybe taken at the contact probe stage (107) and the data are averaged toreduce experimental errors. In still alternative embodiments, repeatmeasurements may be taken at both the hyperspectral stage (102) and thecontact probe stage (107), and the data are averaged to reduceexperimental errors.

In some alternative embodiments, measurements each corresponding to asingle time point are taken at either the hyperspectral stage or thecontact probe stage. In some alternative embodiments, measurements at(corresponding to) a plurality of time points are taken at either thehyperspectral stage or the contact probe stage. For example, a video ofhyperspectral images of a subject may be created by monitoring thepatient or subject over a period of time. The video images may be takenat a time lag separated by milliseconds, seconds, minutes, hours or daysso long as such time lag enables a practitioner to formulate a qualitycharacterization of the subject or patient. Similarly, a video or videorepresentation of measurements of the suspect region may be also createdby monitoring the suspect region over a period of time. The measurementsmay be taken at a time lag separated by milliseconds, seconds, minutes,hours or days so long as such time lag enables a practitioner toformulate a quality characterization of the suspect region. In somealternative embodiments, the subject or the suspected region may beprepared prior to data analysis by the contact probe. For example, thesubject may be dosed with a certain dose of agent or reagent tofacilitate analysis by the contact probe, or alternatively, the suspectregion may be cleaned to allow better data collection.

In some alternative embodiments, the same type of data is collected atthe hyperspectral image stage (102) and at the contact probe stage (107)of a suspect region of the subject. In some alternative embodiments, thedata collected at the contact probe stage (107) of a suspect region onthe subject is different than the type of data that is collected at thehyperspectral image stage (102). For example, in one embodiment the datathat is obtained at the contact probe stage (107) is the type of datadisclosed in U.S. Pat. No. 5,042,494, which is hereby incorporated byreference herein in its entirety. U.S. Pat. No. 5,042,494 discloses amethod and an apparatus for detecting the presence of cancerous tissueusing native visible luminescence. The tissue to be examined is excitedwith a beam of light that causes the tissue to fluoresce over a spectrumof wavelengths. The intensity at which the excited tissue fluoresces canbe measured is either over a spectrum or at a predetermined number ofpreselected wavelengths. One can determine the carcinomatoid status of atissue by determining the wavelength(s) at which maximum intensity(ies)are attained for the tissue in question and by comparing these peakwavelengths, either visually or electronically, to the peakwavelength(s) derived from a known non-cancerous tissue. Alternatively,one can also determine the carcinomatoid status of a tissue by comparingthe luminescence spectrum of the excited tissue with the luminescencespectrum of a known non-cancerous tissue and/or known cancerous tissueor the excitation spectra of the excited tissue with the excitationspectra of known cancerous and/or known non-cancerous tissue, one candetermine the carcinomatoid status of the tissue in question.

In another example, in one alternative embodiment the data obtained atthe contact probe stage (107) is the type of data disclosed in U.S. Pat.No. 5,131,398, which is also hereby incorporated by reference herein inits entirety. U.S. Pat. No. 5,131,398 discloses a method and apparatusfor distinguishing cancerous tumor tissues from benign tumor tissue ornormal tissue using native fluorescence. The tissue to be examined isexcited with a beam of monochromatic light at 300 nanometers (nm). Theintensity of the native fluorescence emitted from the tissue is measuredat 340 and 440 nm. A ratio of the two intensities is then calculated andused as a basis for determining if the tissue is cancerous as opposed tobenign or normal.

In some alternative embodiments, adjustments are made to measurements oranalytical parameter(s) for the contact probes before additionalmeasurements are taken of the suspect region. The additionalmeasurements are then used to improve the result from the earliercontact probe analysis.

In some alternative embodiments, data from the contact probe analysisstep 108 may lead to the conclusion that the initial hyperspectral imageanalysis 103 needs to be modified. Accordingly, modifications ofmeasurements and/or analytical parameters can be made and, based onthese modifications, the hyperspectral data collection 102 and/orhyperspectral image data analysis 103 and/or the additional measurements107 coupled with contact probe analysis step 108 repeated one or moretimes. For example, in some embodiments, contact probe analysis step 108may lead to redefining the suspect region. Thus, in such embodiments,the redefined suspect region may be subject to additional measurements102 and/or 107 with the concomitant review 103 or analysis 108. In someembodiments, modification can be made to the existing suspect regionsuch that the existing suspect region is enlarged, reduced or altered.Review step 108 may be performed using methods similar to those used inthe process for identifying one or more suspect regions. For example,another indication may be identified by comparing measurements obtainedof the suspect region at step 107 with a library of spectral signaturelibrary where the spectral signature library contains a plurality ofspectral signatures each corresponding to a medical condition. Whendifferent types of data are collected for the plurality of regions andfor the suspect region, a different type of signature library can beused during the comparison analysis for the suspect region.

In some embodiments, the additional data obtained with the contactprobes are combined with data from hyperspectral analysis (e.g., ahyperspectral image) for further analysis before a diagnosis of thesubject is made. In some embodiments, such analysis is carried out bycomparing the data from the hyperspectral image or from the contactprobe with existing signatures of signals that correspond to knownillnesses or medical conditions.

It will be understood by one of ordinary skill in the art that anyanalytical methods known in the art may be used to process the dataobtained either at the hyperspectral stage 102 or at the alternative andadditional contact probe stage 107. Exemplary methods for data analysiscan be found hereinbelow.

After a diagnosis of the subject is made (104) based on the firstspectral image, or one or more subsequent images, the subject issubjected to a treatment plan based on that diagnosis (105). Forexample, if the subject is diagnosed with a cancerous lesion that is notreadily apparent to the naked eye but that has boundaries observable inthe hyperspectral medical image, the treatment plan may call for theexcision of the lesion based on the boundaries shown in thehyperspectral medical image.

FIG. 1B illustrates a method 110 of obtaining a hyperspectral medicalimage of a subject for use in diagnosis (for example, at step 103 of themethod of FIG. 1A), according to some embodiments.

First, each of a plurality of regions of the subject is irradiated withlight (111). The regions may collectively represent an area identifiedas being of interest due to the subject's complaints or by visualinspection. Collectively, the regions of the subject can include, forexample, a portion of one of the subject's body parts, an entire bodypart, multiple body parts, or the entire subject. However, eachindividual region may be quite small, e.g., less than 10 centimeters inarea, or less than 1 centimeter in area, or less than 100 millimeters inarea, or less than 10 millimeters in area, or less than 1 millimeter inarea, or less than 100 microns in area. Usefully, each individual regionis sufficiently small to allow resolution of the medical feature ofinterest, that is, so that a specified region containing the medicalfeature can be distinguished from other regions that do not contain thefeature. Different options for the source and spectral content of thelight are described in greater detail below.

Next, light is obtained from the regions of the subject (112). Dependingon the interactions between the regions of the subject and the spectrumof light with which they are irradiated, the light may be reflected,refracted, absorbed, and/or scattered from the regions of the subject.In some embodiments, one or more regions of the subject may even emitlight, e.g., fluoresce or photoluminesce in response to irradiation withthe light. A lens, mirror, or other suitable optical component can beused to obtain the light from the regions of the subject, as describedin greater detail below.

The light obtained from each region is then resolved into acorresponding spectrum (113). For example, the light obtained from eachregion can be passed into a hyperspectral sensor. The hyperspectralsensor includes a diffraction grating or other dispersive opticalcomponent that generates a spatial separation between the light'scomponent wavelengths. This spatial separation allows the relativeintensities of the component wavelengths in the spectrum to be obtainedand recorded, e.g., using a detector such as a charge-coupled device(CCD) or other appropriate sensor that generates a digital signalrepresenting the spectrum. The relative intensities of the componentwavelengths can be calibrated (for example, as described below) toobtain the absolute intensities of those wavelengths, which arerepresentative of the actual physical interaction of the light with thesubject. The calibrated digital signal of each spectrum can be stored,e.g., on tangible computer readable media or in tangible random accessmemory.

A portion of each spectrum is then selected (114). This portionselection can be based on one or more of several different types ofinformation. For example, the portion can be selected based on aspectral signature library (122), which contains information about thespectral characteristics of one or more predetermined medicalconditions, physiological features, or chemicals (e.g., pharmaceuticalcompounds). These spectral characteristics can include, for example,pre-determined spectral regions that are to be selected in determiningwhether the subject has a particular medical condition. Or, for example,the portion can be selected based on a spectral difference between thespectrum of that region and the spectrum of a different region (123).For example, a cancerous region will have a different spectrum than willa normal region, so by comparing the spectra of the two regions thepresence of the cancer can be determined. The portion can also, oralternatively, be selected based on information in other types of imagesof the regions (121). As discussed in greater detail below, visiblelight, LIDAR, THz, and/or other types of images can be obtained of theregions (120). These images may include information that indicates thepresence of a certain medical condition. For example, if a darkenedregion of skin is observed in a visible light image, the portion of thespectrum can be selected so as to include information in some or all ofthe visible light band. Further details on systems and methods ofselecting portions of spectra, and of obtaining other types of images ofthe subject, are provided below.

The selected portions of the spectra are then analyzed (115), forexample, to determine whether the selected portions contain spectralpeaks that match those of a pre-determined medical condition.Optionally, steps 114 and 115 are performed in reverse order. Forexample, the spectra can be compared to that of a pre-determined medicalcondition, and then portions of the compared spectra selected, asdescribed in greater detail below.

A hyperspectral image based on the selected portion of each spectrum isthen constructed (116). The image includes information about therelative intensities of selected wavelengths within the various regionsof the subject. The image can represent the spectral information in avariety of ways. For example, the image may include a two-dimensionalmap that represents the intensity of one or more selected wavelengthswithin each region of the subject. Such image can be monochromatic, withthe intensity of the map at a given region based on the intensity of theselected wavelengths (e.g., image intensity directly proportional tolight intensity at the selected wavelengths). Alternately, the image canbe colorful, with the color of the map at a given region based on theintensity of the selected wavelengths, or indices deducted from theselected wavelengths (for example, a value representative of the ratiobetween the value of a peak in a spectrum and the value of a peak in aspectrum of a medical condition). Although the image may representinformation from one or more non-visible regions of the electromagneticspectrum (e.g., infrared), the image is visible so that it can be viewedby a physician or other interested party.

The hyperspectral image is optionally combined or “fused” with otherinformation about the subject (117). For example, the hyperspectralimage can be overlaid on a conventional visible-light image of thesubject. Also, or alternatively, the image can be combined with theoutput of other types of sensors, such as LIDAR and/or THz sensors.Systems and methods for generating “fused” hyperspectral images aredescribed in greater detail below.

The hyperspectral image, which is optionally fused with otherinformation, is then displayed (118). For example, the image can bedisplayed on a video display and/or can be projected onto the subject,as is described in greater detail in U.S. Provisional Patent ApplicationNo. 61/052,934, filed May 13, 2008 and entitled “Systems and Methods forHyperspectral Medical Imaging Using Real-Time Projection of SpectralInformation,” the entire contents of which are hereby incorporated byreference herein. In embodiments in which the image is projected ontothe subject, the regions of the image corresponding to regions of thesubject are projected directly, or approximately directly, onto thoseregions of the subject. This allows for the concurrent orcontemporaneous inspection of the physical regions of the subject on thesubject as well as on an imaging device such as a computer monitor. Thisfacilitated correlation of those spectral features with physicalfeatures of the subject, thus aiding in the diagnosis and treatment of amedical condition. The delay between obtaining the light and projectingthe image onto the subject and/or onto a computer display may be lessthan about 1 millisecond (ms), less than about 10 ms, less than about100 ms, less than about 1 second, less than about 10 seconds, or lessthan about 1 minute. In some embodiments, the image is a fused imagewhile in other embodiments the image is a hyperspectral image.

In embodiments in which the spectral image is displayed on a videodisplay, the image can be inspected, optionally while the subject isbeing examined, thereby facilitating the procurement of information thatis useful in diagnosing and treating a medical condition. In someembodiments, a conventional visible light image of the regions of thesubject is displayed along with the image containing spectralinformation to aid in the correlation of the spectral features withphysical features of the subject. In some embodiments, the image is bothprojected onto the subject and displayed on a video monitor.

In some embodiments, the hyperspectral image, the raw spectra, and anyother information (such as visible light, LIDAR, and/or THz images) arestored for later processing (119). For example, storing an image of alesion each time the subject is examined can be used to track the growthof the lesion and/or its response to treatment. Storing the spectra canenable other information to be obtained from the spectra at a latertime, as described in greater detail below.

2. Systems for Hyperspectral Medical Imaging

FIG. 2A illustrates an exemplary embodiment of a hyperspectral medicalimaging system 200 that is mounted on a cart 204. The system 200 can bemounted on the cart 204 using, for example, a tripod, a post, a rack, orcan be directly mounted to the cart. The cart 204 includes wheels thatallow system 200 to be readily moved relative to subject 201, thusenabling the system 200 to obtain hyperspectral images of differentparts of the subject's body without requiring the subject to move. Insome embodiments, the system 200 can be moved closer to the subject 201in order to obtain more detailed images of parts of the subject's body(e.g., for diagnostic purposes), and can be moved further away from thesubject 201 in order to obtain a wider view of the subject's body (e.g.,for screening purposes). Alternatively, the system 200 includes zoomingoptics that enable closer or wider views of the subject 201 to be imagedwithout requiring the system to be physically moved closer to or awayfrom the subject. In another embodiment (not shown), the sensor is fixedin place (e.g., is mounted on a tripod), but includes rotatable mirrorsand/or can itself be rotated, enabling different parts of the subject201 to be imaged without moving the sensor relative to the subject, andzooming optics for varying how close a view of the subject is imaged.

The subject 201 is illustrated as standing, but the subject couldgenerally be in any suitable position, for example, lying down, sitting,bending over, etc. In some embodiments, the system 200 is insteadmounted on a scanning bed and/or on an articulated arm. Alternatively,the system 200 can be handheld and include fiber optics that aremanually moved to obtain reflected light from different portions of thesubject.

The system 200 includes an illumination subsystem 210 for irradiatingthe subject 201 with light (illustrated as dashed lines); a sensorsubsystem 230 that includes a hyperspectral sensor (HS Sensor) 231, acamera 280, and a THz sensor 290, a processor subsystem for analyzingthe outputs of the sensor subsystem 230 and generating a fusedhyperspectral image, and a display subsystem 270 that includes a videodisplay 271 for displaying the fused hyperspectral image in real-time,and optionally also includes a projector (not shown) for projecting thefused hyperspectral image onto the subject 201.

FIG. 2B schematically illustrates the components of the hyperspectralmedical imaging system 200 of FIG. 2A, according to some embodiments. InFIG. 2B, the subject is represented as an area 201 that includes aplurality of regions 201′, which are illustrated as a plurality of smallsquares. The area 201 can be one of the subject's body parts or aportion thereof (e.g., a selected area of the subject's skin), can bemultiple body parts or portions thereof, or can even be the entiresubject. The plurality of regions 201′ are subsets of area 201. Theregions 201′ need not be directly adjacent one another, and need not besquare, or even regularly shaped. The regions 201′ collectivelyrepresent a sampling of the area 201 that is to be characterized. In theillustrated embodiment, the regions 201′ are organized into rows 202 andcolumns 203 of regions. The subject is, of course, not considered to bepart of the imaging system.

As discussed above, the hyperspectral imaging system 200 includes anillumination subsystem 210, a sensor subsystem 230, a processorsubsystem 250, and a display subsystem 270. The processor subsystem 250is in operable communication with each of the illumination, sensor, anddisplay subsystems, and coordinates the operations of these subsystemsin order to irradiate the subject, obtain spectral information from thesubject, construct an image based on the spectral information, anddisplay the image. Specifically the illumination subsystem 210irradiates with light each region 201′ within area 201 of the subject,which light is represented by the dashed lines. The light interacts withthe plurality of regions 201′ of the subject. The sensor subsystem 230collects light from each region of the plurality of regions 201′ of thesubject, which light is represented by the dotted lines. Thehyperspectral sensor 231 within sensor subsystem 230 resolves the lightfrom each region 201′ into a corresponding spectrum, and generates adigital signal representing the spectra from all the regions 201′. Theprocessor subsystem 250 obtains the digital signal from the sensorsubsystem 230, and processes the digital signal to generate ahyperspectral image based on selected portions of the spectra that thedigital signal represents. The processor optionally fuses thehyperspectral image with information obtained from the camera 280 (whichcollects light illustrated as dash-dot lines) and/or the THz sensor 290(which collects light illustrated as dash-dot-dot lines) The processorsubsystem 250 then passes that image to projection subsystem 270, whichdisplays the image.

Each of the subsystems 210, 230, 250, and 270 will now be described ingreater detail.

A. Illumination Subsystem

Illumination subsystem 210 includes a light source 212, a lens 211, andpolarizer 213.

The light source 212 generates light having a spectrum that includes aplurality of component wavelengths. The spectrum can include componentwavelengths in the X-ray band (in the range of about 0.01 nm to about 10nm); ultraviolet (UV) band (in the range of about 10 nm to about 400nm); visible band (in the range of about 400 nm to about 700 nm); nearinfrared (NIR) band (in the range of about 700 nm to about 2500 nm);mid-wave infrared (MWIR) band (in the range of about 2500 nm to about 10μm); long-wave infrared (LWIR) band (in the range of about 10 μm toabout 100 μm); terahertz (THz) band (in the range of about 100 μm toabout 1 mm); or millimeter-wave band (also referred to as the microwaveband) in the range of about 1 mm to about 300 mm, among others. The NIR,MWIR, and LWIR are collectively referred to herein as the infrared (IR)band. The light can include a plurality of component wavelengths withinone of the bands, e.g., a plurality of wavelengths in the NIR band, orin the THz. Alternately, the light can include one or more componentwavelengths in one band, and one or more component wavelengths in adifferent band, e.g., some wavelengths in the visible, and somewavelengths in the IR. Light with wavelengths in both the visible andNIR bands is referred to herein as “VNIR.” Other useful ranges mayinclude the region 1,000-2,500 nm (shortwave infrared, or SWIR).

The light source 212 includes one or more discrete light sources. Forexample, the light source 212 can include a single broadband lightsource, a single narrowband light source, a plurality of narrowbandlight sources, or a combination of one or more broadband light sourceand one or more narrowband light source. By “broadband” it is meantlight that includes component wavelengths over a substantial portion ofat least one band, e.g., over at least 20%, or at least 30%, or at least40%, or at least 50%, or at least 60%, or at least 70%, or at least 80%,or at least 90%, or at least 95% of the band, or even the entire band,and optionally includes component wavelengths within one or more otherbands. A “white light source” is considered to be broadband, because itextends over a substantial portion of at least the visible band. By“narrowband” it is meant light that includes components over only anarrow spectral region, e.g., less than 20%, or less than 15%, or lessthan 10%, or less than 5%, or less than 2%, or less than 1%, or lessthan 0.5% of a single band. Narrowband light sources need not beconfined to a single band, but can include wavelengths in multiplebands. A plurality of narrowband light sources may each individuallygenerate light within only a small portion of a single band, buttogether may generate light that covers a substantial portion of one ormore bands, e.g., may together constitute a broadband light source.

One example of a suitable light source 212 is a diffused lighting sourcethat uses a halogen lamp, such as the Lowel Pro-Light Focus Flood Light.A halogen lamp produces an intense broad-band white light which is aclose replication of daylight spectrum. Other suitable light sources 212include a xenon lamp, a hydrargyrum medium-arc iodide lamp, and/or alight-emitting diode. In some embodiments, the light source 212 istunable. Other types of light sources are also suitable. In someembodiments, a standard or custom filter is used to balance the lightintensities at different wavelengths to raise the signal level ofcertain wavelength. For examples, a blue filter can be used to increasethe relative intensity of blue lights for which the sensor has a lowsensitivity. Such filters could also be used with a foreoptic (e.g., thelens) of the hyperspectral sensor.

Depending on the particular light source 212 used, the relativeintensities of the light's component wavelengths are uniform (e.g., aresubstantially the same across the spectrum), or vary smoothly as afunction of wavelength, or are irregular (e.g., in which somewavelengths have significantly higher intensities than slightly longeror shorter wavelengths), and/or can have gaps. Alternatively, the lightcan include one or more narrow-band spectra in regions of theelectromagnetic spectrum that do not overlap with each other.

The light from light source 212 passes through lens 211, which modifiesthe focal properties of the light (illustrated as dashed lines) so thatit illuminates regions 201′ of the subject. In some embodiments, lens211 is selected such that illumination subsystem 210 substantiallyuniformly irradiates regions 201′ with light. That is, the intensity oflight at one region 201′ is substantially the same as the intensity oflight at another region 201′. In other embodiments, the intensity of thelight varies from one region 201′ to the next.

The light then passes through optional polarizer 213, which removes anylight that does not have a selected polarization. Polarizer 213 can be,for example, a polarizing beamsplitter or a thin film polarizer. Thepolarization can be selected, for example, by rotating polarizer 213appropriately.

Illumination subsystem 210 irradiates regions 201′ with light ofsufficient intensity to enable sensor subsystem 230 to obtainsufficiently high quality spectra from those regions 201′, that is, thata spectrum with a sufficient signal-to-noise ratio can be obtained fromeach region 201′ to be able to obtain medical information about eachregion 201′. However, in some embodiments, ambient light, such asfluorescent, halogen, or incandescent light in the room, or evensunlight, is a satisfactory source of light. In such embodiments, theillumination subsystem 210 is not activated, or the system may not eveninclude illumination system 210. Sources of ambient light typically donot communicate with the processing subsystem 250, but instead operateindependently of system 200.

The light from illumination subsystem 210 (illustrated as the dashedlines in FIG. 2B) interacts with the plurality of regions 201′ withinarea 201. The interaction between the light and each region 201′ dependson the particular physiological structure and characteristics of thatregion. The particular interactions between the light and eachindividual irradiated region of the subject impart a spectral signatureonto the light obtained from that region. This spectral signature can beused to obtain medical information about the subject. Specifically,different regions interact differently with the light depending on thepresence of, for example, a medical condition in the region, thephysiological structure of the region, and/or the presence of a chemicalin the region. For example, fat, skin, blood, and flesh all interactwith various wavelengths of light differently from one another.Similarly, a given type of cancerous lesion interacts with variouswavelengths of light differently from normal skin, from non-cancerouslesions, and from other types of cancerous lesions. A given chemicalthat is present (e.g., in the blood, or on the skin) interacts withvarious wavelengths of light differently from other types of chemicals.Thus, the light obtained from each irradiated region of the subject hasa spectral signature based on the characteristics of the region, whichsignature contains medical information about that region.

For example, the structure of skin, while complex, can be approximatedas two separate and structurally different layers, namely the epidermisand dermis. These two layers have very different scattering andabsorption properties due to differences of composition. The epidermisis the outer layer of skin. It has specialized cells called melanocytesthat produce melanin pigments. Light is primarily absorbed in theepidermis, while scattering in the epidermis is considered negligible.For further details, see G. H. Findlay, “Blue Skin,” British Journal ofDermatology 83(1), 127-134 (1970), the entire contents of which arehereby incorporated by reference herein.

The dermis has a dense collection of collagen fibers and blood vessels,and its optical properties are very different from that of theepidermis. Absorption of light of a bloodless dermis is negligible.However, blood-borne pigments like oxy- and deoxy-hemoglobin and waterare major absorbers of light in the dermis. Scattering by the collagenfibers and absorption due to chromophores in the dermis determine thedepth of penetration of light through skin.

In the visible and near-infrared (VNIR) spectral range and at lowintensity irradiance, and when thermal effects are negligible, majorlight-tissue interactions include reflection, refraction, scattering andabsorption. For normal collimated incident radiation, the regularreflection of the skin at the air-tissue interface is typically onlyaround 4%-7% in the 250-3000 nanometer (nm) wavelength range. Forfurther details, see R. R. Anderson and J. A. Parrish, “The optics ofhuman skin,” Journal of Investigative Dermatology 77(1), 13-19 (1981),the entire contents of which are hereby incorporated by referenceherein. When neglecting the air-tissue interface reflection and assumingtotal diffusion of incident light after the stratum corneum layer, thesteady state VNIR skin reflectance can be modeled as the light thatfirst survives the absorption of the epidermis, then reflects backtoward the epidermis layer due the isotropic scattering in the dermislayer, and then finally emerges out of the skin after going through theepidermis layer again.

Using a two-layer optical model of skin, the overall reflectance can bemodeled as:R(λ)=T _(E) ²(λ)R _(D)(λ),where T_(E)(λ) is the transmittance of epidermis and R_(D)(λ) is thereflectance of dermis. The transmittance due to the epidermis is squaredbecause the light passes through it twice before emerging out of skin.Assuming the absorption of the epidermis is mainly due to the melaninconcentration, the transmittance of the epidermis can be modeled as:T _(E)(λ)=exp(d _(E) c _(m) m(λ)),where d_(E) is the depth of the epidermis, c_(m) is the melaninconcentration and m(λ) is the absorption coefficient function formelanin. For further details, see S. L. Jacques, “Skin optics,” OregonMedical Laser Center News Etc. (1988), the entire contents of which arehereby incorporated by reference herein.

The dermis layer can be modeled as a semi-infinite homogeneous medium.The diffuse reflectance from the surface of dermis layer can be modeledas:

${{R_{D}(\lambda)} = {\exp\left( \frac{- A}{\sqrt{3\left( {1 + {{\mu_{s}(\lambda)}/{\mu_{a}(\lambda)}}} \right)}} \right)}},$where constant A is approximately 7-8 for most soft tissues, andμ_(a)(λ) is the overall absorption coefficient function of the dermislayer. For further details, see S. L. Jacques, “Diffuse reflectance froma semi-infinite medium,” Oregon Medical Laser News Etc. (1999), theentire contents of which are hereby incorporated by reference herein.

The term μ_(a)(λ) can be approximated as:μ_(a)(λ)=c _(o) o(λ)+c _(h) h(λ)+c _(w) w(λ),where c_(o), c_(h), and c_(w) are the concentrations of oxy-hemoglobin,deoxy-hemoglobin and water, respectively, while o(λ), h(λ), and w(λ) arethe absorption coefficient functions of oxy-hemoglobin,deoxy-hemoglobin, and water, respectively. For further details, see S.Wray et al., “Characterization of the near infrared absorption spectraof cytochrome aa3 and haemoglobin for the non-invasive monitoring ofcerebral oxygenation,” Biochimica et Biophysica Acta 933(1), 184-192(1988), the entire contents of which are hereby incorporated byreference herein.

The scattering coefficient function for soft tissue can be modeled as:μ_(s)(λ)=aλ ^(−b),where a and b depend on the individual subject and are based, in part,on the size and density of collagen fibers and blood vessels in thesubject's dermis layer.

From the above equations, for a fixed depth of epidermis layer, the skinreflectance R(λ) can be modeled as a function f of seven parameters:R(λ)=f(a,b,c _(m) ,c _(o) ,c _(h) ,c _(w),λ)where a, b, c_(m), c_(o), c_(h), and c_(w), are as described above. Theskin reflectance R(λ) may also depend on other variables not listedhere. For example, long wavelengths (e.g., in the MWIR, FIR, or THzbands) may interact weakly with the surface of the skin and interactstrongly with fat, flesh, and/or bone underlying the skin, and thereforevariables other than those discussed above may be relevant.

The value of the skin's reflectance as a function of wavelength, R(λ),can be used to obtain medical information about the skin and itsunderlying structures. For example, when skin cancers like basal cellcarcinoma (BCC), squamous cell carcinoma (SCC), and malignant melanoma(MM) grow in the skin, the molecular structure of the affected skinchanges. Malignant melanoma is a cancer that begins in the melanocytespresent in the epidermis layer. For further details, see “Melanoma SkinCancer,” American Cancer Society (2005), the entire contents of whichare hereby incorporated by reference herein. Most melanoma cells producemelanin that in turn changes the reflectance characteristics as afunction of wavelength R(λ) of the affected skin. Squamous and basalcells are also present in the epidermis layer. The outermost layer ofthe epidermis is called the stratum corneum. Below it are layers ofsquamous cells. The lowest part of the epidermis, the basal layer, isformed by basal cells. Both squamous and basal cell carcinomas producecertain viral proteins that interact with the growth-regulating proteinsof normal skin cells. The abnormal cell growth then changes theepidermis optical scattering characteristics and consequently the skinreflectance properties as a function of wavelength R(λ). Thus,information about different skin conditions (e.g., normal skin, benignskin lesions and skin cancers) can be obtained by characterizing thereflectance R(λ) from the skin. This can be done, for example, using thesensor subsystem 230 and processor subsystem 250, as described ingreater detail below.

B. Sensor Subsystem

As illustrated in FIG. 2B, the sensor subsystem 230 includes ahyperspectral sensor 231 that obtains light from each region 201′ andresolves that light into a corresponding spectrum; a THz sensor 290 thatobtains THz light from each region 201′ and generates an intensity maprepresenting the intensity of THz light reflected from each region 201′;and a camera 280 that obtains visible light from each region 201′ andgenerates an intensity map representing the intensity of visible lightfrom each region 201′ (e.g., a conventional photographic image). Thehyperspectral sensor 231, THz sensor 290, and camera 280 will each bediscussed in turn.

It should be understood that the THz sensor and camera are optionalfeatures of the sensor subsystem 230, and that the sensor subsystem 230may also or alternatively include other types of sensors, such as aLIDAR sensor (laser detection and ranging), a thermal imaging sensor, amillimeter-wave (microwave) sensor, a color sensor, an X-ray sensor, aUV (ultraviolet) sensor, a NIR (near infrared) sensor, a SWIR (shortwave infrared) sensor, a MWIR (mid wave infrared) sensor, or a LWIR(long wave infrared) sensor. Other types of sensors can also be includedin sensor subsystem 230, such as sensors capable of making non-opticalmeasurements (e.g., molecular resonance imaging, nuclear magneticresonance, a dynamic biomechanical skin measurement probe). Some sensorsmay obtain information in multiple spectral bands. In some embodiments,one or more sensors included in the sensor subsystem 230 arecharacterized by producing an intensity map of a particular type ofradiation from the regions 201′, as opposed to producing a spectrum fromeach region 201′, as does the hyperspectral sensor 231. In someembodiments, one or more sensors included in the sensor subsystem 230 inaddition to the hyperspectral sensor produce a spectrum that can beanalyzed.

In one example, a LIDAR sensor can obtain 3D relief and digitizedrenderings of the regions 201′, which can augment lesion analysis.Physicians conventionally touch a subject's skin while developing theirdiagnosis, e.g., to determine the physical extent of a lesion based onits thickness. A LIDAR sensor, if used, records the topography of alesion with an accuracy far exceeding that possible with manualtouching. A LIDAR sensor functions by scanning a pulsed laser beam overa surface, and measuring the time delay for the laser pulses to returnto the sensor, for each point on the surface. The time delay is relatedto the topographical features of the surface. For medical imaging, theintensity and color of the laser beam used in the LIDAR sensor isselected so that it does not injure the subject. Conventionally, LIDARis performed at a relatively large distance from the object beingscanned. For example, LIDAR systems can be mounted in an airplane andthe topology of the earth measured as the airplane passes over it. WhileLIDAR sensors that operate at close ranges suitable for medicalenvironments are still in development, it is contemplated that such asensor can readily be incorporated into sensor subsystem 230. Someexamples of sensors suitable for producing 3D topological images of asubject include, but are not limited to, the VIVID 9i or 910 Non-Contact3D Digitizers available from Konica Minolta Holdings, Inc., Tokyo,Japan, and the Comet IV, Comet 5, T-Scan, and T-Scan 2 scannersavailable from Steinbichler Optotechnik GmbH, Neubeuern, Germany.

i. Hyperspectral Sensor

Referring to FIG. 2B, the hyperspectral sensor 231 includes a scanmirror 232, a polarizer 233, a lens 234, a slit 235, a dispersive optic236, a charge-coupled device (CCD) 237, a sensor control subsystem 238,and a storage device 239. It should be understood that the optics can bedifferently arranged than as illustrated in FIG. 2B (e.g., the opticscan be in a different order than shown, optics can be eliminated, and/oradditional optics provided).

The scan mirror 232 obtains light from one row 202 of the regions 201′at a time (illustrated as dotted lines in FIG. 2B), and directs thatlight toward the other optics in the sensor 231 for spectral analysis.After obtaining light from one row 202, the scan mirror 232 then rotatesor otherwise moves in order to obtain light from a different row 202.The scan mirror 232 continues this rotation until light has beensequentially obtained from each row 202. Mechanisms other than scanmirrors can be used to scan sequential rows of regions 201′ of thesubject, such as the focal plane scanner described in Yang et al., “ACCD Camera-based Hyperspectral Imaging System of Stationary and AirborneApplications,” Geocarto International, Vol. 18, No. 2, June 2003, theentire contents of which are incorporated by reference herein. In someembodiments (not shown), the hyperspectral sensor 231 insteadsequentially obtains light from rows 202 by moving relative the subject,or by the subject moving relative to the sensor.

The light then passes through optional polarizer 233, which removes anylight that does not have a selected polarization. Polarizer 233 can be,for example, a polarizing beamsplitter or a thin film polarizer, with apolarization selected, for example, by rotating polarizer 233appropriately. The polarization selected by polarizer 233 can have thesame polarization, or a different polarization, than the polarizationselected by polarizer 213. For example, the polarization selected bypolarizer 233 can be orthogonal (or “crossed”) to the polarizationselected by polarizer 213. Crossing polarizers 213 and 233 can eliminatesignal contributions from light that does not spectrally interact withthe subject (and thus does not carry medical information about thesubject), but instead undergoes a simple specular reflection from thesubject. Specifically, the specularly reflected light maintains thepolarization determined by polarizer 213 upon reflection from thesubject, and therefore will be blocked by crossed polarizer 233 (whichis orthogonal to polarizer 213). In contrast, the light that spectrallyinteracts with the subject becomes randomly depolarized during thisinteraction, and therefore will have some component that passes throughcrossed polarizer 233. Reducing or eliminating the amount of specularlyreflected light that enters the hyperspectral sensor 231 can improve thequality of spectra obtained from the light that spectrally interactedwith the subject and thus carries medical information.

In crossed-polarizer embodiments, the intensity of the light that passesthrough polarizer 233 (namely, the light that becomes depolarizedthrough interaction with the subject) has somewhat lower intensity thanit would if polarizers were excluded from the system. The light can bebrought up to a satisfactory intensity, for example, by increasing theintensity of light from illumination subsystem 210, by increasing theexposure time of CCD 237, or by increasing the aperture of lens 234. Inan alternative embodiment, polarizers 213 and 233 are not used, andspecular reflection from the subject is reduced or eliminated by using a“diffuse” light source, which generates substantially uniform light frommultiple angles around the subject. An example of a diffuse light sourceis described in U.S. Pat. No. 6,556,858, entitled “Diffuse InfraredLight Imaging System,” the entire contents of which are incorporated byreference herein.

The lens 234 obtains light from polarizer 233, and suitably modifies thelight's focal properties for subsequent spectral analysis.

The optional slit 235 then selects a portion of the light from the lens234. For example, if the scan mirror 232 obtains light from more thanone row 202 of regions 201′ at a time, and the slit 235 can eliminatelight from rows other than a single row of interest 202.

The light is then directed onto dispersive optic 236. The dispersiveoptic 236 can be, for example, a diffractive optic such as transmissiongrating (e.g., a phase grating or an amplitude grating) or reflectivegrating, prism, or other suitable dispersive optic. The dispersive optic236 spatially separates the different component wavelengths of theobtained light, allowing the intensity of each of the componentwavelengths (the spectrum) to be obtained for each region 201′ of theselected row 202.

FIG. 3A schematically illustrates the resolution of the spectrum of eachregion 201′ in a row 202 into an exemplary “hyperspectral data plane”305. The plane 305 includes a plurality of columns 301′, each of whichincludes the spectrum of a corresponding region 201′. As FIG. 3Aillustrates, the intensity of the spectrum within each column 301′varies as a function of wavelength. This intensity variation is a resultof the light's wavelength-dependent interaction with the correspondingregion 201′ of the subject, and thus contains medical information aboutthat region 201′. For example, using the model described above, thespectrum can be modeled as a wavelength-dependent reflectance R(λ) thatis a function of several variables, e.g., the concentrations of melanin,oxy-hemoglobin, deoxy-hemoglobin and water. In the illustratedembodiment, a dark color at a given wavelength means less reflection oflight from the region 201′ (e.g., strong absorption of that wavelengthby the region 201′, such as due to a high concentration of melanin) anda light color at a given wavelength means more reflection of light fromthe region 201′ (e.g., weak absorption of that wavelength by the region201′, such as due to a low concentration of melanin) Thus, in FIG. 3Athe plane 305 indicates that the left-most columns 301′ had a relativelyhigh reflection at long wavelengths, which reflects the fact that theleft-most regions 201′ of row 202 contain different medical informationthan the right-most regions 201 of row 202.

Under control of the sensor control subsystem 238, the CCD 237 sensesand records the intensity of each of the component wavelengths (thespectrum) from each region 201′ of row 202 the form of a digital signal,such as a hyperspectral data plane. In some embodiments, the sensorcontrol subsystem 238 stores the plane in storage device 239. Storagedevice 239 can be volatile (e.g., RAM) or non-volatile (e.g., a harddisk drive). The hyperspectral sensor 231 then sequentially obtainsadditional planes 305 for the other rows 202, and storing thecorresponding planes 305 in storage 239.

FIG. 3B illustrates a “hyperspectral data cube” 306 that thehyperspectral sensor 231 constructs using the planes 305 obtained foreach of the rows 202 within area 201. The cube 306 includes a spectrum307 corresponding to each region 201′. The spectra are stored within athree-dimensional volume, in which two of the axes represent the x- andy-coordinates of the regions 201′, and the third axis represents thewavelengths within the corresponding spectra. The intensity at aparticular point within the cube 306 represents the intensity of aparticular wavelength (λ) at a particular region 201′ having coordinates(x, y).

The hyperspectral sensor 231 stores cube 306 in storage device 239, andthen passes the cube 306 to processor subsystem 250. In otherembodiments, the sensor control subsystem 238 provides hyperspectraldata planes to the processor subsystem 250, which then constructs,stores, and processes the hyperspectral data cubes 306. The spectracorresponding to the regions 201′ can, of course, be stored in any othersuitable format, or at any other suitable location (e.g., storedremotely).

The CCD can include, but is not limited to, a Si CCD, a InGaAs detector,and a HgCdTe detector. Suitable spectral ranges in some embodiments is0.3 microns to 1 micron, 0.4 micron to 1 micron, 1 micron to 1.7microns, or 1.3 microns to 2.5 microns. In some embodiments the detectorcontains between 320 and 1600 spatial pixels. In other embodiments, theCCD has more or less spatial pixels. In some embodiments, the detectorhas a field of view between 14 degrees and 18.4 degrees. In someembodiments the CCD 237 samples at a rate of between 3 nm and 10 nm. Insome embodiments, the CCD samples between 64 and 256 spectral bands. Ofcourse, it is expected over time that improved CCDs or other types ofsuitable detectors will be devised and any such improved detector can beused.

Within hyperspectral sensor 231, the CCD 237 is arranged at a fixeddistance from the dispersive optic 236. The distance between the CCD 237and the dispersive optic 236, together with the size of the sensorelements that make up the CCD 236, determines (in part) the spectralresolution of the hyperspectral sensor 231. The spectral resolution,which is the width (e.g., full width at half maximum, or FWHM) of thecomponent wavelengths collected by the sensor element, is selected so asto be sufficiently small to capture spectral features of medicalconditions of interest. The sensed intensity of component wavelengthsdepends on many factors, including the light source intensity, thesensor element sensitivity at each particular component wavelength, thereflectance or transmittance of different sensor components such as scanmirror, polarizer, lens, and dispersive optic, and the exposure time ofthe sensor element to the component wavelength. These factors areselected such that the sensor subsystem 230 is capable of sufficientlydetermining the intensity of component wavelengths that it candistinguish the spectral features of medical conditions of interest.

The sensor control subsystem 238 can be integrated with the CCD 237, orcan be in operable communication with the CCD 237. Collectively, thedispersive optic 236 and CCD 237 form an imaging spectrometer (which canalso include other components). Note that the efficiency of a dispersiveoptic and the sensitivity of a CCD can be wavelength-dependent. Thus,the dispersive optic and CCD can be selected so as to have satisfactoryperformance at all of the wavelengths of interest to the measurement(e.g., so that together the dispersive optic and CCD allow a sufficientamount of light to be recorded from which a satisfactory spectrum can beobtained).

One example of a suitable hyperspectral sensor 231 is the AISAhyperspectral sensor, which is an advanced imaging spectrometermanufactured by Specim (Finland). The AISA sensor measureselectromagnetic energy over the visible and NIR spectral bands,specifically from 430 nm to 910 nm. The AISA sensor includes a “pushbroom” type of sensor, meaning that it scans a single line at a time,and has a spectral resolution of 2.9 nm and a 20 degree field of vision.An AISA hyperspectral sensor does not include an integrated polarizer233 as is illustrated in FIG. 2B, but such a polarizer can optionally beincluded external to the AISA hyperspectral sensor.

Other types of sensors can also be used, that collect light from theregions 201′ in other orders. For example, light can be obtained and/orspectrally resolved concurrently from all regions 201′. Or, for example,the light from each individual region 201′ can be obtained separately.Or, for example, the light from a subset of the regions can be obtainedconcurrently, but at a different time from light from other subsets ofthe regions. Or, for example, a portion of the light from all theregions can be obtained concurrently, but at a different time from otherportions of the light from all the regions (for example, the intensityof a particular wavelength from all regions can be measuredconcurrently, and then the intensity of a different wavelength from allregions can be measured concurrently). In some embodiments, light isobtained from a single row 202 at a time, or a single column 203 at atime.

For example, some embodiments include a liquid crystal tunable filter(LCTF) based hyperspectral sensor. An LCTF-based sensor obtains lightfrom all regions 201′ at a time, within a single narrow spectral band ata time. The LCTF-based sensor selects the single band by applying anappropriate voltage to the liquid crystal tunable filter, and recordinga map of the reflected intensity of the regions 201′ at that band. TheLCTF-based sensor then sequentially selects different spectral bands byappropriately adjusting the applied voltage, and recording correspondingmaps of the reflected intensity of the regions 201′ at those bands.Another suitable type of sensor is a “whisk-broom” sensor thatconcurrently collects spectra from both columns and rows of regions 201′in a pre-defined pattern. Not all systems use a scan mirror 232 in orderto obtain light from the subject. For example, an LCTF-based sensorconcurrently obtains light from all regions 201′ at a time, so scanningthe subject is not necessary.

Suitable modifications for adapting the embodiments described herein foruse with other types of hyperspectral sensing schemes will be apparentto those skilled in the art.

ii. Camera

As FIG. 2B illustrates, the sensor subsystem 230 also includes a camera280. The camera 280 can be, for example, a conventional video or digitalcamera that produces a conventional visible-light image of the regions201′.

The camera 280 includes a lens 281, a CCD 282, and an optional polarizer283. The lens 281 can be a compound lens, as is commonly used inconventional cameras, and may have optical zooming capabilities. The CCD282 can be configured to take “still” pictures of the regions 201′ witha particular frequency, or alternatively can be configured to take alive video image of the regions 201′.

The camera 280, the hyperspectral sensor 231 and/or the THz sensor 290can be co-bore sighted with each other. By “co-bore sighted” it is meantthat the center of each sensor/camera points to a common target. Thiscommon focus permits the output of each sensor/camera to bemathematically corrected so that information obtained from eachparticular region 201′ with a particular sensor/camera can be correlatedwith information obtained from that particular region 201′ with all ofthe other sensors/cameras. In one example, the camera and sensor(s) areco-bore sighted by using each camera/sensor to obtain an image of a grid(e.g., a transparent grid fastened to the subject's skin) The grid marksin each respective image can be used to mathematically correlate thedifferent images with each other (e.g., to find a transform that allowsfeatures in one image to be mapped directly onto corresponding featuresin another image). For example, a hyperspectral image, which may have arelatively low spatial resolution, can be fused with a high spatialresolution visible light image, yielding a hyperspectral image ofsignificantly higher resolution than it would have without fusion.

One example of useful medical information that can be obtained fromvisible-light images includes geometrical information about medicalconditions, such as lesions. Lesions that have irregular shapes, andthat are larger, tend to be cancerous, while lesions that have regularshapes (e.g., are round or oval), and that are smaller, tend to bebenign. Geometrical information can be included as another criterion fordetermining whether regions of a subject contain a medical condition.

One example of a suitable camera 280 is a Nikon D300 camera, which is asingle-lens reflex (SLR) digital camera with 12.3 megapixel resolutionand interchangeable lenses allowing highly detailed images of thesubject to be obtained.

iii. THz Sensor

The development of THz sensors for use in medical imaging is an area ofmuch active research. Among other things, THz imaging is useful becauseTHz radiation is not damaging to tissue, and yet is capable of detectingvariations in the density and composition of tissue. For example, somefrequencies of terahertz radiation can penetrate several millimeters oftissue with low water content (e.g., fatty tissue) and reflect back.Terahertz radiation can also detect differences in water content anddensity of a tissue. Such information can in turn be correlated with thepresence of medical conditions such as lesions.

A wide variety of THz sensors exist that are suitable for use in sensorsubsystem 230. In some embodiments, THz sensor 290 includes a THzemitter 291, a THz detector 292, and a laser 293. THz emitter 291 can,for example, be a semiconductor crystal with non-linear opticalproperties that allow pulses of light from laser 293 (e.g., pulses withwavelengths in the range of 0.3 μm to 1.5 μm) to be converted to pulseswith a wavelength in the THz range, e.g., in the range of 25 GHz to 100THz, or 50 GHz to 84 THz, or 100 GHz to 50 THz. The emitter 291 can bechosen from a wide range of materials, for example, LiO₃, NH₄H₂PO₄, ADP,KH₂PO₄, KH₂AsO₄, quartz, AlPO₄, ZnO, CdS, GaP, GaAs, BaTiO₃, LiTaO₃,LiNbO₃, Te, Se, ZnTe, ZnSe, Ba₂NaNb₅O₁₅, AgAsS₃, proustite, CdSe,CdGeAs₂, AgGaSe₂, AgSbS₃, ZnS, DAST (4-N-methylstilbazolium), or Si.Other types of emitters can also be used, for example, photoconductiveantennas that emit radiation in the desired frequency range in responseto irradiation by a beam from laser 293 having a different frequency andupon the application of a bias to the antenna. In some embodiments,laser 293 is a Ti:Sapphire mode-locked laser generating ultrafast laserpulses (e.g., having temporal duration of less than about 300 fs, orless than about 100 fs) at about 800 nm.

The THz radiation emitted by emitter 291 is directed at the subject, forexample, using optics specially designed for THz radiation (notillustrated). In some embodiments, the THz radiation is focused to apoint at the subject, and the different regions of the subject arescanned using movable optics or by moving the subject. In otherembodiments, the THz radiation irradiates multiple points of the subjectat a time. The THz radiation can be broadband, e.g., having a broadrange of frequencies within the THz band, or can be narrowband, e.g.,having only one frequency, or a narrow range of frequencies, within theTHz band. The frequency of the THz radiation is determined both by thefrequency or frequencies of the laser 293 and the non-linear propertiesof the emitter 291.

The THz radiation that irradiates the subject (illustrated by thedash-dot-dot lines in FIG. 2B) can be reflected, refracted, absorbed,and/or scattered from the regions of the subject. THz radiation tends topenetrate deeply into tissue, and to partially reflect at interfacesbetween different types of tissue (which have different indices ofrefraction). As different portions of the THz radiation interact withdifferent types of tissue, and reflect from different buried featuresunder the surface of the subject's skin, those portions collect bothspectral information about the composition of the tissue with which theyinteract, as well as structural information about the thicknesses of thedifferent layers of tissue and the speed with which the THz radiationpassed through the tissue.

The THz detector 292 detects the THz radiation from the subject. As isknown in the art, conventional THz detectors can use, for example,electro-optic sampling or photoconductive detection in order to detectTHz radiation. In some embodiments, the THz detector 292 includes aconventional CCD and an electro-optical component that converts thatconverts the THz radiation to visible or NIR radiation that can bedetected by the CCD. The THz signal obtained by the THz detector 292 canbe resolved in time and/or frequency in order to characterize thecomposition and structure of the measured regions of the subject.

Some embodiments use a pump-delayed probe configuration in order toobtain spectral and structural information from the subject. Suchconfigurations are known in the art.

One example of a suitable THz imaging system is the T-Ray 400 TD-THzSystem, available from Picometrix, LLC, Ann Arbor, Mich. Another THzimaging system is the TPI Imaga 1000 available from Teraview, Cambridge,England. For a survey of other currently available systems and methodsfor THz imaging, see the following references, the entire contents ofeach of which are incorporated herein by reference: “Imaging withterahertz radiation,” Chan et al., Reports on Progress in Physics 70(2007) 1325-1379; U.S. Patent Publication No. 2006/0153262, entitled“Terahertz Quantum Cascade Layer;” U.S. Pat. No. 6,957,099, entitled“Method and Apparatus for Terahertz Imaging;” and U.S. Pat. No.6,828,558, entitled “Three Dimensional Imaging.”

In some embodiments, the THz sensor generates an intensity map of thereflection of THz radiation from the subject. In other embodiments, theTHz sensor generates a THz spectral data cube, similar to thehyperspectral data cube described above, but instead containing a THzspectrum for each region of the subject. The spectra contained in such acube can be analyzed similarly using techniques analogous to those usedto analyze the hyperspectral data cube that are described herein.

C. Contact Probes

Further encompassed is a contact probe for further characterizing asubject that has been characterized by hyperspectral image techniques.For purposes of clarity, the part of the instrument that is associatedwith hyperspectral image data measurements is called a hyperspectralimage module. Accordingly, the part of the instrument comprising one ormore contact probes is called a contact probe module. It will beunderstood that there may be functional or structural overlap betweenthe two types of modules. In general, a contact probe module providesfurther characterization of the subject in addition to anycharacterization by the hyperspectral image module. For example, afterhyperspectral images of one or more patients are analyzed, a suspectregion may be identified and subject to further analysis by a contactprobe.

In some embodiments, the contact probe module provides the same type ofcharacterization as the hyperspectral image module but at higherresolution due to a closer proximity between the contact module and thesubject than the hyperspectral image module and the subject. In someembodiments, a contact probe module provides additional characterizationthat has not been performed by the hyperspectral image module. In someembodiments, both the hyperspectral image module and the contact probemodule obtain radiometric measurements (e.g., radiance, radiant flux, orradiant intensity; including absorption and diffraction electromagneticradiation in the visible and invisible range) and/or photometricmeasurements (e.g., luminance, luminous flux, or luminous intensity).For example, in some embodiments, a contact probe measures signalscorresponding to any optical properties of a normal tissue and a tumortissue, including but not limited to, a Melanin signal, a signalreflecting blood flow and oxy to deoxy hemoglobin ratios, a signalreflecting relative decrease/increase in epidermal and dermalcomponents, a signal reflecting cellular interactions, a signalreflecting protein-based interactions and/or a signal accompanying otherbiochemical changes.

In some embodiments, the contact probe module may be used to measuremorphological changes. For example, the three dimensional contour of asuspect region may be measured by a contact probe and compared withcontour features of known types of tissues.

In some embodiments, signals other than radiometric and photometricsignals are measured by the contact probe module. For examplemeasurements of a chemical signal, an electrical signal, a magneticsignal, a thermal signal, a mechanical signal, a radiometric signal, aphotometric signal, or an optical signal can be obtained. For example,native visible luminescence and/or native fluorescence signals may beused to distinguish a cancer tissue versus a non-cancer tissue. Certainnative fluorophores present in virtually all tissues, e.g., tryptophan,NADH, collagen, elastin, and flavins, have been shown to opticallychange between malignant and non-malignant states of a tissue. Theabsorption and fluorescence spectral features of these fluorophores arewell characterized in the art. These fluorophores are among many otherswith similar properties from which a selection may be made. Changes influorescence patterns arise from a number of these fluorophores and theefficiency to emit and absorb light at a particular wavelength. Some ofthese molecules are likely involved in protein synthesis, electrontransport chain and production of energy in the body, e.g., tryptophan,NADH, and flavins. Collagen, for example, is manufactured in the body topatch wounds and mend broken bones. Elastin protein fibers aremanufactured in the body to give muscles strength. By monitoring some ofthese molecules, such as tryptophan, NAPH, and flavins, one can detectchanges in the underlying metabolism, electron transport, and molecularactivity in malignant and non-malignant states of tissues.

In some embodiments, the contact probe is configured for measuring aplurality of types of signals associated with the suspect region. Forexample, a contact probe can obtain measurements of an inherent propertyof the suspect region, such as conductivity, internal pressure, and/ortemperature. A contact probe can measure how the suspect region respondsto an external perturbation; for example, how a type of tissue (e.g.,skin) responds to external pressure by measuring the elasticity of thesuspect region, or how a type of tissue responds to optical signals byobtaining an optical property such as a refractive index. In someembodiments, the suspect region is prepared or processed beforeadditional measurements are taken. For example, the suspect region maybe cleaned

In some embodiments, the contact probe module is used to measure thesuspect region at a single time point by, for example, taking a highresolution hyperspectral image of the suspect region. In someembodiments, the contact probe module is used to capture measurements ofthe suspect region at a plurality of time points. For example, thecontact probe can be a video probe that takes continuous images of thesuspect region.

In some embodiments, the contact probe module is a separate device thatis independent and separate from the hyperspectral image module. In someembodiments, a contact probe module comprises a physical device thatdirectly or nearly directly contacts the surface of the portion of theobject that it measures. Generally, the contact probe module isphysically connected to and communicates with the hyperspectral imagemodule. In some embodiments, the connection is achieved by an electronicconnection through conventional wired circuit(s) or through fiberoptical connection(s). In some embodiments, the connection between thecontact probe module and the hyperspectral image module disclosed hereinis achieved through wireless connection. Alternatively, the contactprobe module can be connected to any part of the system disclosedherein. In some embodiments, a contact probe of a handheld contact probemodule can be placed directly or nearly directly on a subject. Forexample, a contact probe is a fiber-optic probe consisting of a firstplurality of illumination fibers and a second plurality of signalcollection fibers. In some embodiments, the contact probe is positionedclosely to a patient without directly contacting the patient.Measurements collected from the patient are then transmitted to othercomponents of the hyperspectral image system disclosed herein. In someembodiments, the distance between the patient and the contact probe is 1mm or less, 2 mm or less, 5 mm or less, 10 mm or less, 20 mm or less, 50mm or less, 100 mm or less, 250 mm or less, or 500 mm or less.

As depicted in FIG. 2C, an exemplary contact probe module 260 comprisesa probe head unit 262, a handle unit 265, and a connector unit 268. Theprobe head unit, handle unit and the connector unit may be linkedtogether by cables and/or electrical/optical/wireless connections. Itwill be understood that any type of electrical/optical connectors thatcan enable electrical or optical communications between the contactprobe module and the other parts of the hyperspectral image systemdisclosed herein may be used to establish such connection. In someembodiments, contact probe module 260 comprises only a probe head unit262 and a handle unit 265. In some embodiments, a wireless connection isused to transfer signal and/or measurement data from the contact probemodule to the rest of the hyperspectral image system disclosed herein.

Probe Head Unit 262.

In some embodiments, the probe head unit 262 is used to survey a suspectregion of the subject that is being observed. The suspect region can be1 cm² or smaller, 2 cm² or smaller, 5 cm² or smaller, 8 cm² or smaller,10 cm² or smaller, 15 cm² or smaller, 20 cm² or smaller, 30 cm² orsmaller, 50 cm² or smaller, 75 cm² or smaller, 100 cm² or smaller, 150cm² or smaller, 200 cm² or smaller, or 500 cm² or smaller. For example,a hyperspectral image of a baby may suggest that the baby's left cheekneeds to be further examined and accordingly a contact probe may be usedto further analyze the baby's left cheek. The contact probes of theinstant application may adopt different shapes and sizes when analyzingdifferent samples. The probe head can have a surface area that is 0.25cm² or smaller, 0.5 cm² or smaller, 1 cm² or smaller, 2 cm² or smaller,5 cm² or smaller, 8 cm² or smaller, 10 cm² or smaller, 15 cm² orsmaller, 20 cm² or smaller, 30 cm² or smaller, 50 cm² or smaller, 75 cm²or smaller, 100 cm² or smaller, 150 cm² or smaller, 200 cm² or smaller,or 500 cm² or smaller. The probe head can be round, triangular,circular, oval, tubular, or of any suitable shapes and sizes. Forexample, for the baby's cheek, a small flat-headed contact probe may beused to avoid damaging the delicate skin of the baby. In otherembodiments, a contact probe may have a thin protruded shape such as aneedle, an elongated probe, etc. Such probes may be used to measure, forexample, the inside of the mouth of a patient to provide additional datainformation to facilitate a diagnosis.

Probes of various capacities may be used as contact probes inconjunction with the hyperspectral image device disclosed herein. Acontact probe may be designed for collecting radiometric or photometricmeasurements. In some embodiments, a contact probe utilizes the sametype of sensor device as the hyperspectral image module. The physicalproximity between a contact probe and a subject allows the contact probeto collect signals that contain less interference from, for example,ambient light. A contact probe can direct the signals towards theexisting sensor system of the hyperspectral image system that eventuallyrenders a hyperspectral image of the suspect region that is at a higherresolution. In some embodiments, the contact probe even shares a sensorsystem with other part of the hyperspectral image system disclosedherein.

Alternatively, a contact probe that measures a different type of signalsfrom those measured by the hyperspectral image module may be used toprovide additional characterization of the suspect regions.

In some embodiments, the sensor device for qualifying the signalsmeasured by the contact probe module is located on the contact probemodule. In some embodiments, the sensor device for qualifying thesignals measured by the contact probe module is bundled with theexisting sensor device in the hyperspectral image component disclosedherein. Contact probe module functions as a conduit means through whichsignals from the suspect region are passed to the existing sensor devicein the hyperspectral image module or an independent shared sensordevice. The sensor device may comprise multiple types of sensors for alltypes of signals obtained by both the contact probe module and thehyperspectral image component.

Types of probes that may be used in the probe head unit include, but arenot limited to, an optical probe for visible light, ultra violet light,or infra red light such as a fiber-optic probe; an electrical probe suchas a voltmeter probe, an oscilloscope probe, a passive scope probe, a Z₀probe, an active scope probe, a differential probe, a current probe, anear-field probe; a temperature probe, or any type of probe that iscapable of receiving, detecting, or recording signals from the suspectregion.

In some embodiments, fiber-optic probes may be used. A typicalfiber-optic probe comprises a plurality of illumination fibers and aplurality of signal collection fibers. In various embodiments there canbe 5 or more illumination fibers, 10 or more each illumination fibers,15 or more each illumination fibers, 20 or more each illuminationfibers, 30 or more each illumination fibers, 50 or more eachillumination fibers, or 100 or more each illumination fibers in afiber-optic probe. Similarly, a fiber-optic probe may have 5 or moresignal collection fibers, 10 or more each signal collection fibers, 15or more each signal collection fibers, 20 or more each signal collectionfibers, 30 or more each signal collection fibers, 50 or more each signalcollection fibers, or 100 or more each signal collection fibers. It willbe understood by one skilled in the art that, there can be anycombination of illumination fibers and signal collection fibers. Thenumber of the different types of optic-fibers may be related to thephysical design (e.g., shape and size) of the contact probe, or thefunctional design (e.g., surface or deep tissue detection) of the probe.An exemplary fiber-optic probe design that can be used as a contactprobe is described in Tumeh et al., 2007, “Differentiation of Vascularand Non-Vascular Skin Spectral Signatures Using in vivo HyperspectralRadiometric Imaging,” Cancer Biology & Therapy 6:3, 447-453, which ishereby incorporated by reference herein in its entirety. In anotherexample, a fiber-optic based probe is employed to direct ultravioletillumination onto a tissue specimen and to collect the fluorescentresponse radiation. The response radiation is observed at multiplewavelengths. The intensities of different wavelength can be normalizedand used to determine a score using the ratios in a linear discriminantanalysis (LDA). See U.S. Pat. No. 7,127,282 to Nordstrom et al. issuedon Oct. 24, 2006 (optical methods and systems for rapid screening of thecervix), which is hereby incorporated by reference herein in itsentirety.

Additional exemplary types of contact probes that can be used include,but are not limited to, those described, for example, in U.S. Pat. No.5,413,108 (Method and Apparatus for Mapping a Tissue Sample for andDistinguishing Different Regions Thereof Based on LuminescenceMeasurements of Cancer-Indicative Native Fluorophore); U.S. Pat. No.5,036,853 (Physiological Probe); U.S. Pat. No. 5,792,053 (Hybrid Probefor Tissue Type Recognition); U.S. Pat. No. 5,830,146 (Sheathed Probesfor Tissue type Recognition); U.S. Pat. No. 5,842,995 (SpectroscopicProbe for In vivo Measurement of Raman Signals); U.S. Pat. No. 5,941,834(Sheath for a Side View Probe); U.S. Pat. No. 5,953,306 (Micro NeedleProbe Apparatus having Probes Cantilevered Over Respective ElectronicCircuits, Moving Medium Memory Device Including Same and Method ofMaking Same); U.S. Pat. No. 6,477,132 (Probe and InformationRecording/Reproduction Apparatus Using the Same); U.S. Pat. No.7,185,440 (Sensing Contact Probe); U.S. Pat. No. 7,310,547 (FluorescentFiberoptic Probe for Tissue Health Discrimination) each of which ishereby incorporated by reference herein in its entirety, as well asUnited States Patent Publication Nos. 20030215791 A120050254709 A1, eachof which is hereby incorporated by reference herein in its entirety.

It will be understood by one of ordinary skill in the art that the probehead unit 262 may be designed to incorporate interchangeability suchthat multiple types of probe head units may be interchangeably attachedto the handle unit 265 such that more than one type of signals may beobtained and measured for further analyses.

In addition to the native fluorophores that are inherent to the humanbody, synthetic and exogenous fluorophores may be temporarilyadministered to a subject in order to facilitate data collection. Suchadministration can be, for example, through oral means, injection orsimply by rubbing the reagent onto the suspect region.

More on additional devices that may be used for data collection bycontact probes. For example, additional light source, additionalreagents, additional equipment for temporary or permanently storingmeasurements collected from the suspect region.

Handle Unit 265.

Preferably, a contact probe module is a handheld device. In someembodiments, the handle unit 265 has an elongated shape to fit the handof a user operating the contact probe module. The handle unit 265 may becylindrical or tubular. In some embodiments, handle unit 265 has ahollow space that along one of its dimensions (e.g., the longestdimension) such that cables or wires for electrical and/or opticalconnection between the probe head unit 262 and the hand unit 265 may beaccommodated in the hollow space. In some embodiments, adaptor motif(such as a screw-type adaptor or a magnet-type adaptor) is found at oneor both ends of the hollow space for connecting the handle unit 265 tothe probe head unit 262 and to the connector unit 268. In someembodiments, the adaptor motif for the probe head unit 262 is designedto receive multiple types of probe head units. Any type or number ofcontact probes discussed hereinabove may be attached to the probe head.

In other embodiments, the handle unit 265 comprises a groove-likestructure for accommodating the cables or wires for electrical and/oroptical connection between the probe head unit 262 and the hand unit265. The groove-like structure may be embedded inside the handle unit265 or on the surface of in the handle unit 265.

In some embodiments, the handle unit 265 is ergonomically designed suchthat it conveniently and comfortably fits the hand of an operator of thecontact probe module. In some embodiments, the handle unit 265 is madeof rigid materials such as plastic, wood, metal, stainless steel,aluminum or an aluminum alloy. In other embodiments, the handle unit 265is made of flexible materials such as a rubberlike material, a rubberderivative, silicone rubber, or an elastomer, natural rubber, vulcanizedrubber, a butadiene-styrene polymer such as GR-S, neoprene, nitrilerubbers, butyl, polysulfide rubber, ethylene-propylene rubber,polyurethane rubber, silicone rubber, gutta-percha, and/or balata. Instill other embodiments, the handle unit 265 is made of rigid andflexible material, for example, a rigid tubular structure coated with aflexible rubber-like material for comfort and easy gripping.

Connector Unit 268.

In preferred embodiments, connector unit 268 links the contact probemodule with the rest of the disclosed hyperspectral image apparatus. Insome embodiments, the contact probe is an integrated part of the sensordevice described herein above. For example, a handheld contact probemodule is permanently attached to some embodiments of the disclosed thehyperspectral image device. Alternatively, a handheld contact probe canoperate as a separate and removable module that may be transferred fromone image device to another image device. Such transferable portabilitycan be enabled by standard connection through a connector unit 268. Forexample, connector unit 268 can be a standard electrical/opticalconnector plug which can be plugged into a receiving outlet on any partof the hyperspectral medical imaging system 200.

In addition to the units described above, there may be additional unitsin the contact probe module including but not limited to an additionallight source, an additional display device, a data window, or anoptional memory device for storing data.

Data transfer between the contact probe module and the other part of thesystem occurs through wired connection or wirelessly. It is possiblethat the data collected by the contact probe unit may be storedtemporarily on the contact probe unit (e.g. in a USB memory key, plug indevice, etc.) before being transferred to the hyperspectral dataprocessing module for further processing.

D. Processor Subsystem

Referring to FIG. 2B, the processor subsystem 250 includes a storagedevice 252, a spectral calibrator 253, a spectral analyzer 254, an imageconstructor 256, and a power supply 258. The processor subsystem is inoperable communication with the illumination subsystem 210, the sensorsubsystem 230, and the display subsystem 270.

The processor subsystem 210 instructs illumination subsystem 210 toirradiate the regions 201′ of the subject. Optionally, the processorsubsystem 210 controls the polarization selected by polarizer 213, e.g.,by instructing illumination subsystem 210 to rotate polarizer 213 to aparticular angle corresponding to a selected polarization.

The processor subsystem 250 instructs hyperspectral sensor 231, in thesensor subsystem 230, to obtain spectra of the regions 201′. Theprocessor subsystem 250 can provide the hyperspectral sensor 231 withinstructions of a variety of parameter settings in order to obtainspectra appropriately for the desired application. These parametersinclude exposure settings, frame rates, and integration rates for thecollection of spectral information by hyperspectral sensor 231.Optionally, the processor subsystem 250 also controls the polarizationselected by polarizer 233, e.g., by instructing hyperspectral sensor 231to rotate polarizer 233 to a particular angle corresponding to aselected polarization.

The processor subsystem 250 then obtains from hyperspectral sensor 231the spectra, which may be arranged in a hyperspectral data plane orcube. The processor subsystem 250 also obtains from sensor subsystem 230information from any other sensors, e.g., camera 280 and THz sensor 290.The processor subsystem 250 stores the spectra and the information fromthe other sensors in storage device 252, which can be volatile (e.g.,RAM) or non-volatile (e.g., a hard disk drive).

The spectral calibrator 253 then calibrates the spectra stored in thehyperspectral data cube, and optionally the images obtained from othersensors in sensor subsystem 230, using a spectral calibration standardand techniques known in the art. In some instances the spectralcalibration standard comprises a spatially uniform coating thatdiffusely reflects a known percentage of light (e.g., any percentage inthe range between 1% or less of light up through and including 99% ormore of light). In some embodiments, the output of a sensor can becalibrated by obtaining an image of the spectral calibration standardusing that sensor. Because the percentage of light reflected from thestandard is known for each wavelength, the responsiveness of the sensorat each wavelength can be accurately determined (e.g., the sensor can becalibrated) by comparing the measured reflection of light from thestandard to the expected reflection of light from the standard. Thisallows the illumination-independent and wavelength-dependent reflectanceof the subject to be measured far more accurately than if a spectralcalibration standard had not been used. In other embodiments, when thelight intensity of the light source and the sensitivity of thehyperspectral sensor are known, the reflectance of the subject can bedirectly calculated without help of a spectral calibration standard.

As described in greater detail below, the spectral analyzer 254 thenanalyzes selected portions of the spectra, and then the imageconstructor 256 constructs a hyperspectral image based on the analyzedspectra. Optionally, the image constructor 256 fuses the hyperspectralimage with other information about the subject, e.g., images obtainedusing camera 280 and/or THz sensor 290.

The power supply 258 provides power to the processor subsystem 250, andoptionally also provides power to one or more other components ofhyperspectral imaging system 200. The other components of thehyperspectral imaging system 200 can alternately have their own powersupplies. In some embodiments, for example where imaging system 200 isintended to be portable (e.g., can be carried by hand and/or is usableoutside of a building), the power supply 258 and/or other power suppliesin the system 200 can be batteries. In other embodiments, for examplewhere imaging system 200 is fixed in place, or where imaging system isintended to be used inside of a building, the power supply 258 and/orother power supplies in the system 200 can obtain their power from aconventional AC electrical outlet.

The spectral analyzer 254 and the image constructor 256 will now bedescribed in greater detail. Then, an exemplary computer architecturefor processor subsystem 250 will be described.

i. Spectral Analyzer

In some embodiments, the spectral analyzer 254 analyzes the spectraobtained from storage 252 by comparing the spectral characteristics of apre-determined medical condition to the subject's spectra within definedspectral ranges. Performing such a comparison only within definedspectral ranges can both improve the accuracy of the characterizationand reduce the computational power needed to perform such acharacterization.

The spectral characteristics of a medical condition, such as particularlesion type, can be determined, for example, by first identifying anactual skin lesion of that type on another subject, for example usingconventional visual examination and biopsy, and then obtaining thewavelength-dependent reflectance R_(SL)(λ) of a representative region ofthat skin lesion. The skin lesion's reflectance R_(SL)(λ) can then bespectrally compared to the wavelength-dependent reflectance of thatsubject's normal skin in the same area of the lesion, R_(NS)(λ), bynormalizing the reflectance of the skin lesion against the reflectanceof normal skin as follows:R _(SL,N)(λ)=R _(SL)(λ)/R _(NS)(λ),where R_(SL,N)(λ) is the normalized reflectance of the skin lesion. Inother embodiments, R_(SL,N)(λ) is instead determined by taking thedifference between R_(SL)(λ) and R_(NS)(λ), or by calculatingR_(SL,N)(λ)=[R_(SL)(λ)−R_(NS)(λ)]/[R_(SL)(λ)+R_(NS)(λ)]. Other types ofnormalization are possible. Note that if there are multiplerepresentative regions of one skin lesion, there will be as manynormalized reflectances of the skin lesion. These normalizedreflectances can be averaged together, thus accounting for the naturalspectral variation among different regions of the lesion. Note also thatbecause of the natural variation in characteristics of normal skin amongindividuals, as well the potential variation in characteristics of aparticular type of lesion among individuals, it can be useful to basethe model of the normalized skin lesion reflectance R_(SL,N)(λ) on theaverage of the reflectances R_(SL)(λ) of many different skin lesions ofthe same type, as well as on the average of the reflectances R_(NS)(λ)of many different types of normal skin (e.g., by obtaining R_(SL,N)(λ)for many different subjects having that lesion type, and averaging theresults across the different subjects).

In one embodiment, in order to determine whether the subject has thetype of skin lesion characterized by R_(SL,N)(λ), the spectral analyzer254 obtains the skin reflectance of each region 201′, R_(region)(λ),from hyperspectral sensor 231 (e.g., in the form of a hyperspectral dataplane or cube). The spectral analyzer 254 then normalizes thereflectance R_(region)(λ) from that region against thewavelength-dependent reflectance of the subject's normal skin in thesame area, R_(NS, Subject)(λ), as follows:R _(region,N)(λ)=R _(region)(λ)/R _(NS,Subject)(λ),where R_(region,N)(λ) is the normalized reflectance of the region. Othertypes of normalization are possible.

In some embodiments, the spectral analyzer 254 analyzes the subjects'spectra by comparing R_(region,N)(λ) to R_(SL,N)(λ). In one simpleexample, the comparison is done by taking the ratioR_(region,N)(λ)/R_(SL,N)(λ), or the differenceR_(SL,N)(λ)−R_(region,N)(λ). The magnitude of the ratio or differenceindicates whether any region has spectral characteristics that matchthat of the lesion. However, while ratios and differences are simplecalculations, the result of such a calculation is complex and requiresfurther analysis before a diagnosis can be made. Specifically, the ratioor subtraction of two spectra, each of which has many peaks, generates acalculated spectrum that also has many peaks. Some peaks in thecalculated spectrum may be particularly strong (e.g., if the subject hasthe medical condition characterized by R_(SL,N)(λ)), but other peaks mayalso be present (e.g., due to noise, or due to some particularcharacteristic of the subject). A physician in the examination roomwould typically find significantly more utility in a simple “yes/no”answer as to whether the subject has a medical condition, than he wouldin a complex spectrum. One method of obtaining a “yes/no” answer is tocalculate whether a peak in the calculated spectrum has a magnitude thatis above or below a predetermined threshold and is present at awavelength that would be expected for that medical condition.

Another way to obtain a “yes/no” answer is to treat R_(region,N)(λ) andR_(SL,N)(λ) as vectors, and to determine the “angle” between thevectors. The angle represents the degree of overlap between the vectors,and thus represents how likely it is that the subject has the medicalcondition. If the angle is smaller than a threshold value, the subjectis deemed have the medical condition; if the angle does not exceed athreshold value, the subject is deemed not to have the medicalcondition. Alternately, based on the value of the angle between thevectors, a probability that the subject has the medical condition can bedetermined.

While hyperspectral imaging can obtain spectra across broad ranges ofwavelengths (e.g., from 400 nm to 2000 nm), and such breadth allows avast amount of medical information to be collected about the subject,most of the spectrum does not contain information relevant to a single,particular medical condition. For example, skin lesion type “A” may onlygenerate a single spectral peak centered at 1000 nm with 50 nm fullwidth at half maximum (FWHM). Of course, most medical conditionsgenerate considerably more complex spectral features. The rest of thepeaks in the spectrum do not contain information about lesion type “A.”Even though they may contain information about many other types ofmedical conditions, these peaks are extraneous to the characterizationof lesion type “A” and can, in some circumstances, make it moredifficult to determine whether the subject has lesion type “A.”

In some embodiments, the spectral analyzer 254 reduces or eliminatesthis extraneous information by comparing R_(region,N)(λ) to R_(SL,N)(λ)only within specified spectral regions that have been identified asbeing relevant to that particular type of skin lesion. Using the exampleabove, where lesion type “A” only generates a single peak at 1000 nmwith 50 nm FWHM, the spectral analyzer 254 compares R_(region,N)(λ) toR_(SL,N)(λ) only at a narrow spectral region centered at 1000 nm (e.g.,a 50 nm FWHM band centered at 1000 nm). For medical conditions thatgenerate more complex spectral features, the spectral analyzer 254 cancompare R_(region,N)(λ) to R_(SL,N)(λ) within other spectral regions ofappropriate width. Such bands can be determined by statisticallyidentifying which spectral features correlate particularly strongly withthe medical condition as compared with other spectral features that alsocorrelate with the medical condition. For example, when calculating theangle between vectors R_(region,N)(λ) and R_(SL,N)(λ), the extraneousinformation can reduce the angle between the vectors, thus suggesting ahigher correlation between R_(region,N)(λ) and R_(SL,N)(λ) than thereactually is for lesion type “A.”

In one example, a particular medical condition has identifiable spectralcharacteristics within a narrow, contiguous wavelength range λ₁-λ₂(e.g., 850-900 nm). The bounds of this range are stored in storage 252,along with the spectral characteristics of the condition within thatrange. To compare the condition's spectral characteristics to those ofthe subject, the spectral analyzer 254 can first select portions of thesubject's hyperspectral data cube that fall within the desiredwavelength range λ₁-λ₂. Multiple spectral regions can also be selected,and need not be contiguous with one another. The unused spectralportions need not be discarded, but can be saved in storage 252 forlater use, as described in greater detail below.

Following the same example, FIG. 4A illustrates the spectral analyzer'sselection of a volume 406 from the subject's hyperspectral data cube 405within the wavelength range λ₁-λ₂ characteristic of the medicalcondition. The boundaries of volume 406 are defined by the x- andy-dimensions of area 201 and by wavelength range λ₁-λ₂. FIG. 4Billustrates a selected volume 406. The intensity distribution at the topface 410 of the volume corresponds to the spectral intensity atwavelength λ₁ of each region 201′ within the area 201, while theintensity distribution at the bottom face (not shown) of the volumecorresponds to the spectral intensity at wavelength λ₂. Thus it can beseen that regions in the lower left corner of the area 201 stronglyinteracted with light at wavelength λ₁, while regions in the upper rightcorner of the area 201 weakly interacted with light at wavelength λ₁.This indicates that the medical condition is present in the regions inthe lower left corner of area 201, but not in the regions in the upperright corner of area 201. While the volume 406 is illustrated ascontiguous, the selected volume of the hyperspectral cube could insteadbe a combination of multiple sub-volumes that are not adjacent to eachother. Within the selected spectral region(s), R_(region,N)(λ) can becalculated and then compared to R_(SL,N)(λ) using the methods describedabove, or any other suitable method.

There are several other different ways to perform such comparisons onlywithin selected spectral regions. For example, for an angle analysis,the vectors R_(Region)(λ) and R_(SL,N)(λ) can be reduced in size toeliminate values corresponding to wavelengths outside of the selectedspectral regions, and the angle analysis performed as above. Or, forexample, values in the vectors R_(Region)(λ) and R_(SL,N)(λ) that falloutside of the selected spectral regions can be set to zero, and theangle analysis performed as above. For other types of comparisons, forexample, ratios or differences, the ratio or difference values that falloutside of the selected spectral regions can simply be ignored.

In other embodiments, the vectors R_(SL,N)(λ) or multiple lesions of thesame medical condition are used to train a classifier using certainpattern recognition methods as explained in detail later, and the vectorR_(Region)(λ) of an un-identified lesion can be input into theclassifier to see the medical condition is present.

The selection scheme illustrated in FIGS. 4A and 4B is a simple examplebased on the characteristics of a single medical condition stored in aspectral signature library. More complicated schemes can also be used.For example, multiple spectral regions can be selected in parallel or insequence based on the spectral characteristics of multiplepre-determined conditions. For example, as noted above, a physician maynot be able to determine through visual inspection whether a lesion isbenign or cancerous. Thus it can be useful for the spectral analyzer 254to select spectral regions based on the spectral characteristics of awide variety of potential conditions.

The skin lesion example is intended to be merely illustrative. Similarprocedures can be used to obtain a wavelength-dependent reflectance R(λ)for a wide variety of medical conditions and/or physiological featuresand/or chemicals. For example, the R(λ) of a subject having thatcondition/feature/chemical can be obtained and then normalized againstthe R(λ) of a subject lacking that condition/feature/chemical. Spectralregions particularly relevant to that condition/feature/chemical can beidentified and used during the comparison of the condition's reflectanceR(λ) to the subject's reflectance, e.g., as described above.

Regardless of the particular form in which the spectral informationabout the medical condition is stored, in some embodiments the processorsubsystem 250 can access a library of spectral information aboutmultiple medical conditions, that can be used to determine whether thesubject has one or more of those conditions. The library can alsoinclude information about each condition, for example, other indicia ofthe condition, possible treatments of the condition, potentialcomplications, etc.

The library can also store biological information about each conditionthat may be useful in determining whether a subject has the condition.For example, skin pigmentation naturally varies from subject to subject,which causes variations in the wavelength-dependent reflectance betweenthose individuals. These variations can complicate the determination ofwhether a particular individual has a condition. The library can includeinformation that enhances the ability of processor subsystem 250 toidentify whether subjects having a particular skin pigmentation have acondition. Portions of the library can be stored locally (e.g., instorage 252) and/or remotely (e.g., on or accessible by the Internet).

In still other embodiments, portions of spectra are selected based oninformation in other images obtained of the regions 201′, e.g., based oninformation in a visible-light image, a LIDAR image, and/or a THz imageof the regions 201′.

The spectral analyzer 254 can operate on an automated, manual, orsemi-manual basis. For example, in an automatic mode, the spectralanalyzer 254 can fully search the spectral library for conditions havingspectral characteristics that potentially match those of one or more ofthe regions 201′. In a semi-manual mode, a sub-class of conditions canbe identified, or even a single condition, of interest, and the spectralanalyzer can analyze the subject's spectra based on the spectralcharacteristics of that condition or conditions. Or, in a manual mode,the spectral analyzer can operate wholly under the control of a human.In some embodiments, “automated” means without human intervention, and“manual” means with human intervention.

ii. Image Constructor

After the spectral analyzer 254 analyzes the spectra, the imageconstructor 256 constructs an image based on the analyzed spectra.Specifically, the image constructor 256 creates a representation (e.g.,a 2D or 3D representation) of information within the spectra. In oneexample, the image constructor 256 constructs a two-dimensionalintensity map in which the spatially-varying intensity of one or moreparticular wavelengths (or wavelength ranges) within the spectra isrepresented by a corresponding spatially varying intensity of a visiblemarker.

FIG. 5 illustrates an image 510 that is based on the spatial variationsin intensity at wavelength λ₁ that are illustrated in FIG. 4B. The image510 includes regions 511, 512, and 513 of increasing intensity,respectively, which represent the magnitude of interaction of differentregions 201′ with light at wavelength λ₄. While FIG. 5 is monochromatic,false colors can also be assigned to represent different intensities orother information. For example, in embodiments in which multiplespectral portions corresponding to multiple potential conditions areselected, spectral portions corresponding to one condition can beassigned one color, and spectral portions corresponding to anothercondition can be assigned a different color, thus allowing areasaffected by the different conditions to be distinguished.

In some embodiments, the image constructor 256 fuses the hyperspectralimage with information obtained from one or more other sensors in sensorsubsystem 230. For example, as illustrated in FIGS. 7A-7C, differentregions of the electromagnetic spectrum contain significantly differentinformation about a subject. FIG. 7A is an image of a subject obtainedin the visible portion of the spectrum (e.g., is a conventional video orphotographic image of the subject). FIG. 7B is an image of the samesubject, but obtained in the thermal portion of the spectrum (e.g., SWIRto MIR). FIG. 7C is another image of the same subject but obtained instill another portion of the spectrum. The different images wereobtained with appropriate conventional sensors that are known in theart, and highlight different aspects of the medical condition of thesubject. By obtaining relevant information in the appropriateelectromagnetic band(s), and combining that information with an imagerepresenting spectral information about the subject such as thatdescribed herein, images can be generated that provide significantlymore detailed information than an image that represents only a singletype of information.

Information from different sensors can be fused with the hyperspectralimage in many different ways. For example, the hyperspectral image canbe scaled to a grey scale or color, and data from another sensor istopographically scaled to form a topographical or contour map. In suchembodiments, the topographical or contour map can be colored based onthe grey scale or color scaled hyperspectral image. Of course, thereverse is also true, where the hyperspectral image is converted to atopographical or contour map and the data from another sensor isnormalized to a color scale or a grey scale which is then used to colorthe topographical or contour map. Usefully, such a combined map canemphasize skin abnormalities that may not be apparent from any onesensor. For example, if one sensor flags a particular region of thescreen with a “red” result, where red represents one end of the dynamicrange of the sensor, and another sensor assigns a dense peak to thissame region, where the peak represents the limits of the dynamic rangeof this independent sensor, the combined image from the two sensors willshow a peak that is colored red. This can aid in pinpointing a region ofinterest.

Information from one or more sensors can be fused with the hyperspectralimage. In some embodiments, information from two or more, three or more,four or more, five or more sensors are fused with the hyperspectralimage into a single image.

In some embodiments, images obtained using different sensors are takenconcurrently, so that the register of such images with respect to theskin of the subject and to each other is known. In some embodiments,such images are taken sequentially but near in time with the assurancethat the subject has not moved during the sequential measurements sothat the images can be readily combined. In some embodiments, a skinregistry technique is used that allows for the images from differentsensors to be taken at different times and then merged together.

Concurrently using different types of sensors provides a powerful way ofobtaining rich information about the subject. Specific types of sensorsand/or data fusion methods can be used to analyze different types oftargets. For example, in remote sensing analysis, a sensor specific forsubmerged aquatic vegetation (SAV) has been employed. Furthermore,normalized difference vegetation index (NDVI) is also developed forbetter representation. Similarly, in medical imaging, specific sensorsmay be used to detect changes in specific types of tissues, substances,or organs. Indices similar to NDVI can also be developed to normalizecertain types of tissues, substances, or organs, either to enhance theirpresence or to reduce unnecessary background noise.

The information obtained by multi-sensor analysis can be integratedusing data fusion methods in order to enhance image quality and/or toadd additional information that is missing in the individual images. Inthe following section on data fusion methods, the term “sensor” meansany sensor in sensor subsystem 230, including hyperspectral sensor 231,THz sensor 290, and camera 280, or any other type of sensor that is usedin sensor subsystem 230.

In some embodiments, information from different sensors are displayed incomplementary (orthogonal) ways, e.g., in a colorful topographical map.In some embodiments, the information from different sensors is combinedusing statistical techniques such as principal component analysis. Insome embodiments, the information from different sensors is combined inan additive manner, e.g., by simply adding together the correspondingpixel values of images generated by two different sensors. Any suchpixel by pixel based combination of the output of different sensors canbe used.

Image fusion methods can be broadly classified into two categories: 1)visual display transforms; and 2) statistical or numerical transformsbased on channel statistics. Visual display transforms involve modifyingthe color composition of an image, e.g., modifying the intensities ofthe bands forming the image, such as red-green-blue (RGB) or otherinformation about the image, such as intensity-hue-saturation (IHS).Statistical or numerical transforms based on channel statistics include,for example, principal component analysis (PCA). Some non-limitingexamples of suitable image fusion methods are described below.

Band Overlay.

band overlay (also known as band substitution) is a simple image fusiontechnique that does not change or enhance the radiometric qualities ofthe data. Band overlay can be used, for example, when the output fromtwo (or more) sensors is highly correlated, e.g., when the sensors areco-bore sighted and the output from each is obtained at approximatelythe same time. One example of band overlay is panchromatic sharpening,which involves the substitution of a panchromatic band from one sensorfor the multi-spectral band from another sensor, in the same region. Thegeneration of color composite images is limited to the display of onlythree bands corresponding to the color guns of the display device(red-green-blue). As the panchromatic band has a spectral range coveringboth the green and red channels (PAN 0.50-0.75 mm; green 0.52-0.59 mm;red 0.62-0.68 mm), the panchromatic band can be used as a substitute foreither of those bands.

High-Pass Filtering Method (HPF).

The HPF fusion method is a specific application of arithmetic techniquesused to fuse images, e.g., using arithmetic operations such as addition,subtraction, multiplication and division. HPF applies a spatialenhancement filter to an image from a first sensor, before merging thatimage with an image from another sensor on a pixel-by-pixel basis. TheHPF fusion can combine both spatial and spectral information using theband-addition approach. It has been found that when compared to the IHSand PCA (more below), the HPF method exhibits less distortion in thespectral characteristics of the data, making distortions difficult todetect. This conclusion is based on statistical, visual and graphicalanalysis of the spectral characteristics of the data.

Intensity-Hue-Saturation (IHS).

IHS transformation is a widely used method for merging complementary,multi-sensor data sets. The IHS transform provides an effectivealternative to describing colors by the red-green-blue displaycoordinate system. The possible range of digital numbers (DNs) for eachcolor component is 0 to 255 for 8-bit data. Each pixel is represented bya three-dimensional coordinate position within the color cube. Pixelshaving equal components of red, green and blue lie on the grey line, aline from the cube to the opposite corner. The IHS transform is definedby three separate and orthogonal attributes, namely intensity, hue, andsaturation. Intensity represents the total energy or brightness in animage and defines the vertical axis of the cylinder. Hue is the dominantor average wavelength of the color inputs and defines thecircumferential angle of the cylinder. It ranges from blue (0/360°)through green, yellow, red, purple, and then back to blue (360/0°).Saturation is the purity of a color or the amount of white light in theimage and defines the radius of the cylinder.

The IHS method tends to distort spectral characteristics, and should beused with caution if detailed radiometric analysis is to be performed.Although IRS 1C LISS III acquires data in four bands, only three bandsare used for the study, neglecting the fourth due to poor spatialresolution. IHS transform can be more successful in panchromaticsharpening with true color composites than when the color compositesinclude near or mid-infrared bands.

Principal Component Analysis (PCA).

PCA is a commonly used tool for image enhancement and the datacompression. The original inter-correlated data are mathematicallytransformed into new, uncorrelated images called components or axes. Theprocedure involves a linear transformation so that the originalbrightness values are re-projected onto a new set of orthogonal axes.PCA is useful for merging images because of it includes reducing thedimensionality of the original data from n to 2 or 3 transformedprincipal component images, which contains the majority of theinformation from the original sensors. For example, PCA can be used tomerge several bands of multispectral data with one high spatialresolution band.

Image fusion can be done in two ways using the PCA. The first method issimilar to IHS transformation. The second method involves a forwardtransformation that is performed on all image channels from thedifferent sensors combined to form one single image file.

Discrete Wavelet Transform (DWT).

The DWT method involves wavelet decomposition where wavelettransformation converts the images into different resolutions. Waveletrepresentation has both spatial and frequency components. Exemplaryapproaches for wavelet decomposition includes the Mallat algorithm,which can use a wavelet function such as the Daubechies functions (db1,db2, . . . ), and the a Trous algorithm, which merges dyadic wavelet andnon-dyadic data in a simple and efficient procedure.

Two approaches for image fusion based on wavelet decomposition are thesubstitution method and the additive method. In the substitution method,after the wavelet coefficients of images from different sensors areobtained, some wavelet coefficients of one image are substituted withwavelet coefficients of the other image, followed by an inverse wavelettransform. In the additive method, wavelet planes of one image areproduced and added to the other image directly, or are added or to anintensity component extracted from the other image. Some embodiments mayinclude a transformation step.

For further details on exemplary image fusion techniques, see thefollowing references, the entire contents of each of which is herebyincorporated by reference herein: H arris et al., 1990, “IHS transformfor the integration of radar imagery with other remotely sensed data,”Photogrammetric Engineering and Remote Sensing, 56(12), 1631-1641; Pholand van Genderen, 1998, “Multisensor image fusion in remote sensing:concepts, methods and applications,” International Journal of RemoteSensing, 19(5), 823-854; Chavez et al., 1991, “Comparison of threedifferent methods to merge multi-resolution and multi-sectoral data:Landsat™ and SPOT Panchromatic,” Photogrammetric Engineering and RemoteSensing, 57(3), 295-303; Pellemans et al., 1993, “Merging multispectraland panchromatic SPOT images with respect to radiometric properties ofthe sensor,” Photogrammetric Engineering and Remote Sensing, 59(1),81-87; Nunez et al., 1999, “Multiresolution based image fusion withadditive wavelet decomposition,” IEEE Transactions on Geoscience andRemote Sensing, 37(3), 1204-1211; Steinnocher, 1997, “Applications ofadaptive filters for multisensoral image fusion,” Proceedings of theInternational Geoscience and Remote Sensing Symposium (IGARASS '97),Singapore, August 1997, 910-912; and Chavez and Kwarteng, 1989,“Extracting spectral contrast in Landsat Thematic Mapper image datausing selective principal component analysis,” PhotogrammetricEngineering and Remote Sensing, 55(3), 339-348.

iii. Processor Subsystem Architecture

FIG. 6 schematically illustrates an exemplary embodiment of processorsubsystem 250. The subsystem 250 includes a computer system 10 having:

-   -   a central processing unit 22;    -   a main non-volatile storage unit 14, for example a hard disk        drive, for storing software and data, the storage unit 14        controlled by storage controller 12;    -   a system memory 36, preferably high speed random-access memory        (RAM), for storing system control programs, data, and        application programs, including programs and data loaded from        non-volatile storage unit 14; system memory 36 may also include        read-only memory (ROM);    -   a user interface 32, including one or more input devices (e.g.,        keyboard 28, a mouse) and a display 26 or other output device;    -   a network interface card 20 (communications circuitry) for        connecting to any wired or wireless communication network 34        (e.g., a wide area network such as the Internet);    -   a power source 24 to power the aforementioned elements; and    -   an internal bus 30 for interconnecting the aforementioned        elements of the system.

Operation of computer 10 is controlled primarily by operating system(control software) 640, which is executed by central processing unit 22.Operating system (control software) 640 can be stored in system memory36. In some embodiments, system memory 36 also includes:

-   -   a file system 642 for controlling access to the various files        and data structures used herein;    -   the spectral calibrator 253 described above, including        calibration information;    -   the spectral analyzer 254 described above;    -   the image constructor 256 described above;    -   the measured hyperspectral cube 644, which includes a plurality        of measured hyperspectral data planes;    -   a spectral library 646;    -   the selected portion of the measured hyperspectral data cube        660;    -   information from one or more other sensors 670; and    -   the hyperspectral image based on the selected portion of the        measured hyperspectral data cube and optionally fused with        information from other sensors 680.

The measured hyperspectral cube 644, spectral library 646, selectedportion 660, information from other sensors, and the (fused)hyperspectral image can be stored in a storage module in system memory36. The measured hyperspectral data cube 644, the portion selectedthereof 660, the information from other sensors 670, and thehyperspectral image need not all be concurrently present, depending onwhich stages of the analysis that processor subsystem 250 has performed.

The system memory 36 optionally also includes one or more of thefollowing modules, which are not illustrated in FIG. 6:

-   -   a fusion module for fusing a hyperspectral image with        information from other sensors;    -   a trained data analysis algorithm for identifying a region of        the subject's skin of biological interest using an image        obtained by the system; for characterizing a region of the        subject's skin of biological interest using an image obtained by        the apparatus; and/or for determining a portion of a        hyperspectral data cube that contains information about a        biological insult in the subject's skin; and    -   a communications module for transmitting “outline” or “shape”        files to a third party, e.g., using network interface card 20.

As illustrated in FIG. 6, computer 10 includes a spectral library 646,which includes profiles 648 for a plurality of medical conditions,“Condition 1” through “Condition M.” The profile 648 for each conditionincludes a set of spectral characteristics 654 that the spectralanalyzer 254 can use to determine whether the region corresponding tothe measured hyperspectral data cube 644 has condition 1. Each profile648 also includes information about that condition 650, e.g.,information about whether the condition is malignant or benign, optionsfor treatment, etc. Each profile 648 also includes biologicalinformation 652, e.g., information that can be used to modify thedetection conditions for subjects of different skin types. In someembodiments, the spectral library 646 is stored in a single database. Inother embodiments, such data is instead stored in a plurality ofdatabases that may or may not all be hosted by the same computer 10. Insuch embodiments, some of the data illustrated in FIG. 6 as being storedin memory 36 is stored on computer systems that are not illustrated byFIG. 6 but that are addressable by wide area network 34.

In some embodiments, the data illustrated in memory 36 of computer 10 ison a single computer (e.g., computer 10) and in other embodiments thedata illustrated in memory 36 of computer 10 is hosted by severalcomputers (not shown). In fact, all possible arrangements of storing thedata illustrated in memory 36 of computer 10 on one or more computerscan be used so long as these components are addressable with respect toeach other across computer network 34 or by other electronic means.Thus, a broad range of computer systems can be used.

While examining a subject and viewing hyperspectral images of thesubject, the physician can optionally provide input to processorsubsystem 250 that modifies one or more parameters upon which thehyperspectral image is based. This input can be provided using inputdevice 28. Among other things, processor subsystem 250 can be instructedto modify the spectral portion selected by spectral analyzer 254 (forexample, to modify a threshold of analytical sensitivity) or to modifythe appearance of the image generated by image constructor 256 (forexample, to switch from an intensity map to a topological rendering).The processor subsystem 250 can be instructed to communicateinstructions to illumination subsystem 210 to modify a property of thelight used irradiate the subject (for example, a spectralcharacteristic, an intensity, or a polarization). The processorsubsystem 250 can be instructed to communicate instructions to sensorsubsystem 230 to modify the sensing properties of one of the sensors(for example, an exposure setting, a frame rate, an integration rate, ora wavelength to be detected). Other parameters can also be modified. Forexample, the processor subsystem 250 can be instructed to obtain awide-view image of the subject for screening purposes, or to obtain aclose-in image of a particular region of interest.

E. Display Subsystem

The display subsystem 270 obtains the hyperspectral image (which isoptionally fused with information from other sensors) from the imageconstructor 256, and displays the image. In some embodiments, thedisplay subsystem 270 includes a video display 271 for displaying theimage and/or a projector 272 for projecting the image onto the subject.In embodiments including a project, the image can be projected such thatrepresentations of spectral features are projected directly onto, orapproximately onto, the conditions or physiological structures thatgenerated those spectral features.

For further details, see U.S. Provisional Patent Application No.61/052,934, filed May 13, 2008 and entitled “Systems and Methods forHyperspectral Medical Imaging Using Real-Time Projection of SpectralInformation.”

Optionally, the display subsystem 270 also displays a legend thatcontains additional information. For example, the legend can displayinformation indicating the probability that a region has a particularmedical condition, a category of the condition, a probable age of thecondition, the boundary of the condition, information about treatment ofthe condition, information indicating possible new areas of interest forexamination, and/or information indicating possible new information thatcould be useful to obtain a diagnosis, e.g., another test or anotherspectral area that could be analyzed.

3. Applications of Hyperspectral Medical Imaging

A hyperspectral image can be used to make a diagnosis while the subjectis being examined, or any time after the image is obtained. However,there are many other potential applications of hyperspectral imaging,some of which are described below.

A. Personalized Database of Spectral Information

As described above, a hyperspectral image is generated by obtainingspectra from the subject, as well as by optionally obtaining the outputof one or more additional sensors. These spectra, the hyperspectralimage, and the output of other sensors constitute a personalizeddatabase of spectral information for a subject. Additional informationcan be added to the database over time, as the subject is subsequentlyexamined using hyperspectral imaging and the results stored in thedatabase.

Among other things, the database can be used to determine spectralchanges in the subject over time. For example, during a firstexamination, a region of the subject's skin may have a particularspectral characteristic. During a later examination, the region may havea different spectral characteristic, representing a change in themedical condition of the skin. It may be that the skin was normal whenit was first examined (e.g., lacked any noteworthy medical conditions)but obtained a medical condition that was observed during the laterexamination. Alternately, it may be that the skin had a medicalcondition when it was first examined, but the medical conditionunderwent a change that was observed during the subsequent examination,or a new medical condition occurred. The changes to the skin itself maybe imperceptible to a physician's eyes, but can be made apparent throughappropriate hyperspectral analysis. Thus, hyperspectral imaging usingthe subject's own skin as a baseline can allow for significantly earlierdetection of medical conditions than would be possible using otherexamination techniques.

FIG. 8A illustrates a method 800 of using a personalized database ofhyperspectral information for a subject, according to some embodiments.First, a first set of hyperspectral data on a region of the subject isobtained (801), e.g., using the methods described herein. By “set ofhyperspectral data” it is meant spectra, hyperspectral images, andsensor outputs relating to a particular region of skin. The first set ofhyperspectral data can be stored in the personalized database ofhyperspectral information for the subject. Optionally, the database alsoincludes hyperspectral information for other subjects.

At some later time, a second set of hyperspectral data on a region ofthe subject is obtained (802). This second set can also be stored in thepersonalized database of hyperspectral information for the subject.

The second set of hyperspectral data is then compared to the first setof hyperspectral data (803). For example, selected portions of the firstset of hyperspectral data can be compared to corresponding selectedportions of the second set of hyperspectral data. As discussed above,differences between spectra of a particular region can represent achange in the medical condition of the region. Optionally, the firstand/or second sets of hyperspectral data are also compared to a spectralsignature library (806) in order to independently determine whethereither of the sets includes information about a medical condition.

A hyperspectral image of the region is then generated based on thecomparison (804), a diagnosis made based on the hyperspectral image(805), and the subject treated appropriately based on the diagnosis(806).

FIG. 8B illustrates one possible format for a database of hyperspectralinformation. Hyperspectral database 844 includes a plurality of subjectrecords 846. There is no limit on the number of subject records 846 thatcan be held in hyperspectral database 844. Database 844 can hold as fewas one subject record 846. More typically, database 844 holds between 1and 100 subject records, more than 100 subject records, more than athousand subject records, more than ten thousand subject records, morethan 100 thousand subject records, or between 1 subject record and onemillion subject records.

Each subject record 846 preferably includes a subject identifier 848. Asthose skilled in the database arts will appreciate, a subject identifier848 need not be explicitly enumerated in certain database systems. Forinstance, in some systems, a subject identifier 848 can simply be asubject record 846 identifier. However, in some embodiments, a subjectidentifier 48 can be a number that uniquely identifies a subject withina health care program.

Each subject record 846 optionally includes a demographiccharacterization 850 of respective subjects. In some embodiments,relevant portions of the demographic characterization 850 can be used inconjunction with the diagnosis to select a treatment regimen for asubject and/or can be used to characterize features that statisticallycorrelate with the development of a medical condition (more below). Thedemographic characterization for a respective subject can include, forexample, the following features of the subject: gender, marital status,ethnicity, primary language spoken, eye color, hair color, height,weight, social security number, name, date of birth, educational status,identity of the primary physician, name of a referring physician, areferral source, an indication as to whether the subject is disabled anda description of the disability, an indication as to whether the subjectis a smoker, an indication as to whether the subject consumes alcohol, aresidential address of the subject, and/or a telephone number of thesubject. In addition, the demographic characterization 850 can include aname of an insurance carrier for an insurance policy held by the subjectand/or a member identifier number for an insurance policy held by thesubject. In some embodiments, the demographic characterization 850 alsoincludes a family medical history, which can be used when diagnosingand/or treating the subject. The family medical history can include, forexample, data such as whether or not a member of the subject's familyhas a particular medical condition.

Subject records 846 also include outputs from sensor subsystem 230 fromdifferent times the subject was examined. For example, subject records846 can include hyperspectral data cubes 852, THz sensor outputs 854,and/or conventional images 856, or the outputs of any other sensors insensor subsystem 230. Subject records 846 also include hyperspectralimages 858, which may or may not be fused with information from othersensors/cameras.

Subject records 846 also include clinical characterizations 860. In someembodiments, clinical characterizations 860 include observations made bya subject's physician on a particular date. In some instances, theobservations made by a physician include a code from the InternationalClassification of Diseases, 9th Revision, prepared by the Department ofHealth and Human Services (ICD-9 codes), or an equivalent, and datessuch observations were made. Clinical characterizations 860 complementinformation found within the hyperspectral data cubes 852, THz sensoroutputs 854, conventional images 856, and/or hyperspectral images 858.The clinical characterizations 860 can include laboratory test results(e.g., cholesterol level, high density lipoprotein/low densitylipoprotein ratios, triglyceride levels, etc.), statements made by thesubject about their health, x-rays, biopsy results, and any othermedical information typically relied upon by a doctor to make adiagnosis of the subject.

Subject records 846 further include diagnosis fields 862. Diagnosisfields 862 represents the diagnosis for the subject on a particulardate, which can be based upon an analysis of the subject's hyperspectraldata cubes 852, THz sensor outputs 854, conventional images 856,hyperspectral images 858, and/or the clinical characterizations 860 ofthe subject.

Subject data records 846 further include a subject treatment history864. Treatment history 864 indicates the treatment given to a subjectand when such treatment was given. Treatment history 864 includes allprescriptions given to the subject and all medical procedures undergoneon the subject. In some embodiments, the medical procedures includeCurrent Procedural Terminology (CPT) codes developed by the AmericanMedical Association for the procedures performed on the subject, and adate such procedures were performed on the subject.

In some embodiments, a subject data record 846 can also include otherdata 866 such as pathology data (e.g., world health organization(classification, tumor, nodes, metastases staging, images), radiographicimages (e.g., raw, processed, cat scans, positron emission tomography),laboratory data, Cerner electronic medical record data (hospital baseddata), risk factor data, access to a clinical reporting and data system,reference to vaccine production data/quality assurance, reference to aclinical data manager (e.g., OPTX), and/or reference to a cancerregistry such as a research specimen banking database.

B. Temporal “Reachback”

The compilation of hyperspectral databases of one or more subjects canalso be useful in characterizing the development over time of medicalconditions. Among other things, as physicians learn new informationabout a condition, previously collected hyperspectral data can bere-analyzed to determine if that data contains information about thatcondition. For example, a physician in 2010 may discover and spectrallycharacterize a new medical condition. The physician can analyzepreviously collected hyperspectral data in a hyperspectral database(e.g., data from one or more subjects between 2008-2010), to determinewhether that data includes information on the new medical condition. Ifthe physician identifies that a subject in the database had thecondition, even though the condition had not been recognized orcharacterized when the data was collected, the subject's data can beanalyzed to characterize changes over time of the medical condition(e.g., using the method in FIG. 8A). The more subjects that haveinformation in the hyperspectral database, and the greater amount oftime that their information is compiled in the database, the greater thechance that the database will include information not only about aparticular medical condition, but also its development over time and itscharacteristics in different types of subjects. The hyperspectraldatabase can, for example, have the format illustrated in FIG. 8B.

FIG. 9 illustrates a method 900 of obtaining temporal information abouta condition, according to some embodiments. First, the spectralcharacteristics of a condition are identified (901), for example, usingtechniques described herein.

Then, previously collected hyperspectral data for one or more subjectsis analyzed to determine whether any of those subjects had thatcondition, even though it may not have been recognized that they had thecondition at the time the data was collected (902). The previouslycollected hyperspectral data can be stored in a hyperspectral database.

The hyperspectral data for each subject having the condition is thenfurther analyzed to determine spectral characteristics associated withdevelopment of the condition (903). For example, characteristics of theearly presence of the condition, trends of growth among differentsubjects, and patterns of growth within a given subject can all becharacterized.

Based on the determination of the spectral characteristics of thecondition in varying stages of growth over time, the condition can thenbe diagnosed in a new subject using hyperspectral imaging (904). The newsubject can then be treated appropriately.

C. Use of Pattern Classification Techniques

Systems and methods for obtaining high resolution images of patient skinhave been disclosed. Such systems and methods include the generation andstorage of images taken using hyperspectral imaging, digitalphotography, LIDAR, and/or terahertz imaging, to name of few possibletechniques. As discussed herein and in related U.S. Patent Application61/052,934, filed May 13, 2008 and entitled “Systems and Methods forHyperspectral Medical Imaging Using Real-Time Projection of SpectralInformation,” which is hereby incorporated by reference herein, the dataobtained from a subject, particularly the subject's skin, can be fusedimages from any of a number of spectral sources (e.g., hyperspectralimaging, digital photography, LIDAR, and/or terahertz imaging, etc.), orunfused images taken from a single source.

Clearly, the amount of data that is taken from a subject is vast. Forinstance, in the case of hyperspectral imaging, a completethree-dimensional data cube containing several megabytes of data andrepresenting a portion of the subject's skin, is generated. Much work isneeded to analyze such spectral data regardless of whether such spectraldata is from discrete spectral sources and represents the fusion ofspectral data from two or more spectral sources. In such analysis, whatis of interest is the identification of regions of the subject's skinthat may have potential biological insult. Examples of biological insultare skin lesions. Of further interest is the characterization of suchbiological insults. Of further interest is the progression of suchbiological insults over time. Advantageously, as disclosed below in moredetail, systems and methods that assist in such analysis are provided.

First, databases storing any of the data observed and measured using themethods disclosed herein may be electronically stored and recalled. Suchstored images enable the identification and characterization of asubject's skin, and any biological insults thereon, over time.

Second, a wide variety of pattern classification techniques and/orstatistical techniques can be used in accordance with the presentdisclosure to help in the analysis. For instance, such patternclassification techniques and/or statistical techniques can be used to(i) assist in identifying biological insults on a subject's skin, (ii)assist in characterizing such biological insults, and (iii) assist inanalyzing the progression of such biological insults (e.g., detectsignificant changes in such lesions over time).

In one embodiment a database of spectral information, which maycollected over time and/or for many different subjects is constructed.This database contains a wealth of information about medical conditions.In the example provided above, a physician is able to obtain informationabout a newly characterized medical condition, from a previouslyobtained set of spectral data. However, in some circumstances,indications of a medical condition may simply go unrecognized byphysicians. Pattern classification is used to mine the database ofspectral information in order to identify and characterize medicalconditions (biological insults) that are characterized by observables.In some examples, such observables are values of specific pixels in animage of a subject's skin, patterns of values of specific groups ofpixels in an image of a subject's skin, values of specific measuredwavelengths or any other form of observable data that is directlypresent in the spectral data and/or that can be derived from thespectral data taken of a subject's skin. In some embodiments, patternclassification techniques such as artificial intelligence are used toanalyze hyperspectral data cubes, the output of other sensors orcameras, and/or hyperspectral images themselves (which may or may not befused with other information).

FIG. 10 illustrates a method of using a database of spectral informationfrom subject having known phenotypes to train a pattern classificationtechnique or a statistical algorithm, referred to herein as a “dataanalysis algorithm.” The trained data analysis algorithm can then beused to diagnose subjects with unknown phenotypes. The data analysisalgorithm is provided with a spectral training set (1001). Exemplarydata analysis algorithms are described below. The spectral training setis a set of spectral information (e.g., hyperspectral data cubes, theoutput of other sensors or cameras, and/or hyperspectral images) whichmay or may not be fused, which contains characterized information). Forinstance, in one example, the spectral data includes information from asingle sensor (e.g., solely a hyperspectral sensor), discreteinformation from multiple sensors, and/or fused information frommultiple sensors from subjects that have a known medical condition.

As is known in the pattern classification arts, such traininginformation includes at least two types of data, for instance data fromsubjects that have one medical condition and data from subjects thathave another medical condition. See, for example, Golub et al., 1999,Science 531, pp. 531-537, which is hereby incorporated by referenceherein, in which several different classifiers were built using atraining set of 38 bone marrow samples, 27 of which were acutelymphoblastic leukemia and 11 of which were acute mycloid leukemia. Oncetrained, a data analysis algorithm can be used to classify new subjects.For instance in the case of Golub et al., the trained data analysisalgorithm can be used to determine whether a subject has acutelymphoblastic leukemia or acute mycloid leukemia. In the presentdisclosure, a data analysis algorithm can be trained to identify,characterize, or discover a change in a specific medical condition, suchas a biological insult in the subject's skin. Based on the spectraltraining set stored, for example in a database, the data analysisalgorithm develops a model for identifying a medical condition such aslesion, characterizing a medical condition such as a lesion, ordetecting a significant change in the medical condition.

In some embodiments, the trained data analysis algorithm analyzesspectral information in a subject, in order to identify, characterize,or discover a significant change in a specific medical condition. Basedon the result of the analysis, the trained data analysis algorithmobtains a characterization of a medical condition (1002) in a subject inneed of characterization. The characterization is then validated (1003),for example, by verifying that the subject has the medical conditionidentified by the trained data analysis algorithm using independentverification methods such as follow up tests or human inspection. Incases where the characterization identified by the trained data analysisalgorithm is incorrectly called (e.g., the characterization provides afalse positive or a false negative), the trained data analysis algorithmcan be retrained with another training set so that the data analysisalgorithm can be improved.

As described in greater detail below, a model for recognizing a medicalcondition can be developed by (i) training a decision rule usingspectral data from a training set and (ii) applying the trained decisionrule to subjects having unknown biological characterization. If thetrained decision rule is found to be accurate, the trained decision rulecan be used to determine whether any other set of spectral data containsinformation indicative of a medical condition. The input to thedisclosed decision rules is application dependent. In some instances,the input is raw digital feed from any of the spectral sources disclosedherein, either singly or in fused fashion. In some instances, the inputto the disclosed decision rules is stored digital feed from any of thespectral sources disclosed herein, either singly or in fused fashion,taken from a database of such stored data. In some embodiment, the inputto a decision rule is an entire cube of hyperspectral data and theoutput is one or more portions of the cube that are of the mostsignificant interest.

For further details on the existing body of pattern recognition andprediction algorithms for use in data analysis algorithms forconstructing decision rules, see, for example, National ResearchCouncil; Panel on Discriminant Analysis Classification and Clustering,Discriminant Analysis and Clustering, Washington, D.C.: National AcademyPress, the entire contents of which are hereby incorporated by referenceherein. Furthermore, the techniques described in Dudoit et al., 2002,“Comparison of discrimination methods for the classification of tumorsusing gene expression data.” JASA 97; 77-87, the entire contents ofwhich are hereby incorporated by reference herein, can be used todevelop such decision rules.

Relevant algorithms for decision rule include, but are not limited to:discriminant analysis including linear, logistic, and more flexiblediscrimination techniques (see, e.g., Gnanadesikan, 1977, Methods forStatistical Data Analysis of Multivariate Observations, New York: Wiley1977; tree-based algorithms such as classification and regression trees(CART) and variants (see, e.g., Breiman, 1984, Classification andRegression Trees, Belmont, Calif.: Wadsworth International Group;generalized additive models (see, e.g., Tibshirani, 1990, GeneralizedAdditive Models, London: Chapman and Hall; neural networks (see, e.g.,Neal, 1996, Bayesian Learning for Neural Networks, New York:Springer-Verlag; and Insua, 1998, Feedforward neural networks fornonparametric regression In: Practical Nonparametric and SemiparametricBayesian Statistics, pp. 181-194, New York: Springer, the entirecontents of each of which are hereby incorporated by reference herein.Other suitable data analysis algorithms for decision rules include, butare not limited to, logistic regression, or a nonparametric algorithmthat detects differences in the distribution of feature values (e.g., aWilcoxon Signed Rank Test (unadjusted and adjusted)).

The decision rule can be based upon two, three, four, five, 10, 20 ormore measured values, corresponding to measured observables from one,two, three, four, five, 10, 20 or more spectral data sets. In oneembodiment, the decision rule is based on hundreds of observables ormore. Observables in the spectral data sets are, for example, values ofspecific pixels, patterns of values of specific groups of pixels, valuesof specific measured wavelengths or any other form of observable datathat is directly present in the spectral data and/or that can be derivedfrom the spectral data. Decision rules may also be built using aclassification tree algorithm. For example, each spectral data set froma training population can include at least three observables, where theobservables are predictors in a classification tree algorithm (morebelow). In some embodiments, a decision rule predicts membership withina population (or class) with an accuracy of at least about at leastabout 70%, of at least about 75%, of at least about 80%, of at leastabout 85%, of at least about 90%, of at least about 95%, of at leastabout 97%, of at least about 98%, of at least about 99%, or about 100%.

Additional suitable data analysis algorithms are known in the art, someof which are reviewed in Hastie et al., supra. Examples of data analysisalgorithms include, but are not limited to: Classification andRegression Tree (CART), Multiple Additive Regression Tree (MART),Prediction Analysis for Microarrays (PAM), and Random Forest analysis.Such algorithms classify complex spectra and/or other information inorder to distinguish subjects as normal or as having a particularmedical condition. Other examples of data analysis algorithms include,but are not limited to, ANOVA and nonparametric equivalents, lineardiscriminant analysis, logistic regression analysis, nearest neighborclassifier analysis, neural networks, principal component analysis,quadratic discriminant analysis, regression classifiers and supportvector machines. Such algorithms may be used to construct a decisionrule and/or increase the speed and efficiency of the application of thedecision rule and to avoid investigator bias, one of ordinary skill inthe art will realize that computer-based algorithms are not required tocarry out the disclosed methods.

I. Decision Trees

One type of decision rule that can be constructed using spectral data isa decision tree. Here, the “data analysis algorithm” is any techniquethat can build the decision tree, whereas the final “decision tree” isthe decision rule. A decision tree is constructed using a trainingpopulation and specific data analysis algorithms. Decision trees aredescribed generally by Duda, 2001, Pattern Classification, John Wiley &Sons, Inc., New York. pp. 395-396, which is hereby incorporated byreference herein. Tree-based methods partition the feature space into aset of rectangles, and then fit a model (like a constant) in each one.

The training population data includes observables associated with amedical condition. Exemplary observables are values of specific pixels,patterns of values of specific groups of pixels, values of specificmeasured wavelengths or any other form of observable data that isdirectly present in the spectral data and/or that can be derived fromthe spectral data. One specific algorithm that can be used to constructa decision tree is a classification and regression tree (CART). Otherspecific decision tree algorithms include, but are not limited to, ID3,C4.5, MART, and Random Forests. CART, ID3, and C4.5 are described inDuda, 2001, Pattern Classification, John Wiley & Sons, Inc., New York.pp. 396-408 and pp. 411-412, the entire contents of which are herebyincorporated by reference herein. CART, MART, and C4.5 are described inHastie et al., 2001, The Elements of Statistical Learning,Springer-Verlag, New York, Chapter 9, the entire contents of which arehereby incorporated by reference herein. Random Forests are described inBreiman, 1999, “Random Forests-Random Features,” Technical Report 567,Statistics Department, U.C. Berkeley, September 1999, the entirecontents of which are hereby incorporated by reference herein.

In some embodiments, decision trees are used to classify subjects usingspectral data sets. Decision tree algorithms belong to the class ofsupervised learning algorithms. The aim of a decision tree is to inducea classifier (a tree) from real-world example data. This tree can beused to classify unseen examples that have not been used to derive thedecision tree. As such, a decision tree is derived from training data.Exemplary training data contains spectral data for a plurality ofsubjects (the training population), each of which has the medicalcondition. The following algorithm describes an exemplary decision treederivation:

Tree(Examples,Class,Features) Create a root node If all Examples havethe same Class value, give the root this label Else if Features is emptylabel the root according to the most common value Else begin Calculatethe information gain for each Feature Select the Feature A with highestinformation gain and make this the root Feature For each possible value,v, of this Feature Add a new branch below the root, corresponding to A =v Let Examples(v) be those examples with A = v If Examples(v) is empty,make the new branch a leaf node labeled with the most common value amongExamples Else let the new branch be the tree created byTree(Examples(v),Class,Features - {A}) End

In general, there are a number of different decision tree algorithms,many of which are described in Duda, Pattern Classification, SecondEdition, 2001, John Wiley & Sons, Inc. Decision tree algorithms oftenrequire consideration of feature processing, impurity measure, stoppingcriterion, and pruning Specific decision tree algorithms include, butare not limited to classification and regression trees (CART),multivariate decision trees, ID3, and C4.5.

In one approach, when a decision tree is used, the members of thetraining population are randomly divided into a training set and a testset. For example, in one embodiment, two thirds of the members of thetraining population are placed in the training set and one third of themembers of the training population are placed in the test set. Thespectral data of the training set is used to construct the decisiontree. Then, the ability for the decision tree to correctly classifymembers in the test set is determined. In some embodiments, thiscomputation is performed several times for a given combination ofspectral data. In each computational iteration, the members of thetraining population are randomly assigned to the training set and thetest set. Then, the quality of the spectral data is taken as the averageof each such iteration of the decision tree computation.

In addition to univariate decision trees in which each split is based ona feature value for a corresponding phenotype represented by thespectral data set, or the relative values of two such observables,multivariate decision trees can be implemented as a decision rule. Insuch multivariate decision trees, some or all of the decisions actuallyinclude a linear combination of feature values for a plurality ofobservables. Such a linear combination can be trained using knowntechniques such as gradient descent on a classification or by the use ofa sum-squared-error criterion. To illustrate such a decision tree,consider the expression:0.04x ₁+0.16x ₂<500Here, x₁ and x₂ refer to two different values for two differentobservables in the spectral data set. Such observables in the spectraldata set can be, for example, values of specific pixels, patterns ofvalues of specific groups of pixels, values of specific measuredwavelengths or any other form of observable data that is directlypresent in the spectral data and/or that can be derived from thespectral data. To poll the decision rule, the values for x₁ and x₂ areobtained from the measurements obtained from the spectra of unclassifiedsubject. These values are then inserted into the equation. If a value ofless than 500 is computed, then a first branch in the decision tree istaken. Otherwise, a second branch in the decision tree is taken.Multivariate decision trees are described in Duda, 2001, PatternClassification, John Wiley & Sons, Inc., New York, pp. 408-409, which ishereby incorporated by reference herein.

Another approach that can be used is multivariate adaptive regressionsplines (MARS). MARS is an adaptive procedure for regression, and iswell suited for the high-dimensional problems involved with the analysisof spectral data. MARS can be viewed as a generalization of stepwiselinear regression or a modification of the CART method to improve theperformance of CART in the regression setting. MARS is described inHastie et al., 2001, The Elements of Statistical Learning,Springer-Verlag, New York, pp. 283-295, which is hereby incorporated byreference in its entirety.

II. Predictive Analysis of Microarrays (PAM)

One approach to developing a decision rule using values for observablesin the spectral data is the nearest centroid classifier. Such atechnique computes, for each biological class (e.g., has lesion, doesnot have lesion), a centroid given by the average values of observablefrom specimens in the biological class, and then assigns new samples tothe class whose centroid is nearest. This approach is similar to k-meansclustering except clusters are replaced by known classes. This algorithmcan be sensitive to noise when a large number of observables are used.One enhancement to the technique uses shrinkage: for each observable,differences between class centroids are set to zero if they are deemedlikely to be due to chance. This approach is implemented in thePrediction Analysis of Microarray, or PAM. See, for example, Tibshiraniet al., 2002, Proceedings of the National Academy of Science USA 99;6567-6572, which is hereby incorporated by reference herein in itsentirety. Shrinkage is controlled by a threshold below which differencesare considered noise. Observables that show no difference above thenoise level are removed. A threshold can be chosen by cross-validation.As the threshold is decreased, more observables are included andestimated classification errors decrease, until they reach a bottom andstart climbing again as a result of noise observables—a phenomenon knownas overfitting.

III. Bagging, Boosting, and the Random Subspace Method

Bagging, boosting, the random subspace method, and additive trees aredata analysis algorithms known as combining techniques that can be usedto improve weak decision rules. These techniques are designed for, andusually applied to, decision trees, such as the decision trees describedabove. In addition, such techniques can also be useful in decision rulesdeveloped using other types of data analysis algorithms such as lineardiscriminant analysis.

In bagging, one samples the training set, generating random independentbootstrap replicates, constructs the decision rule on each of these, andaggregates them by a simple majority vote in the final decision rule.See, for example, Breiman, 1996, Machine Learning 24, 123-140; and Efron& Tibshirani, An Introduction to Boostrap, Chapman & Hall, New York,1993, the entire contents of which are hereby incorporated by referenceherein.

In boosting, decision rules are constructed on weighted versions of thetraining set, which are dependent on previous classification results.Initially, all features under consideration have equal weights, and thefirst decision rule is constructed on this data set. Then, weights arechanged according to the performance of the decision rule. Erroneouslyclassified biological samples get larger weights, and the next decisionrule is boosted on the reweighted training set. In this way, a sequenceof training sets and decision rules is obtained, which is then combinedby simple majority voting or by weighted majority voting in the finaldecision rule. See, for example, Freund & Schapire, “Experiments with anew boosting algorithm,” Proceedings 13th International Conference onMachine Learning, 1996, 148-156, the entire contents of which are herebyincorporated by reference herein.

To illustrate boosting, consider the case where there are two phenotypesexhibited by the population under study, phenotype 1 (e.g., sick), andphenotype 2 (e.g., healthy). Given a vector of predictor observables(e.g., a vector of values that represent such observables) from thetraining set data, a decision rule G(X) produces a prediction taking oneof the type values in the two value set: {phenotype 1, phenotype 2}. Theerror rate on the training sample is

$\overset{\_}{err} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\;{I\left( {y_{i} \neq {G\left( x_{i} \right)}} \right)}}}$where N is the number of subjects in the training set (the sum total ofthe subjects that have either phenotype 1 or phenotype 2). For example,if there are 49 subjects that are sick and 72 subjects that are healthy,N is 121. A weak decision rule is one whose error rate is only slightlybetter than random guessing. In the boosting algorithm, the weakdecision rule is repeatedly applied to modified versions of the data,thereby producing a sequence of weak decision rules G_(m)(x), m=1, 2, .. . , M. The predictions from all of the decision rules in this sequenceare then combined through a weighted majority vote to produce the finaldecision rule:

${G(x)} = {{sign}\left( {\sum\limits_{m = 1}^{M}\;{\alpha_{m}{G_{m}(x)}}} \right)}$

Here α₁, α₂, . . . , α_(M) are computed by the boosting algorithm andtheir purpose is to weigh the contribution of each respective decisionrule Gm(x). Their effect is to give higher influence to the moreaccurate decision rules in the sequence.

The data modifications at each boosting step consist of applying weightsw₁, w₂, . . . , w_(n) to each of the training observations (x_(i),y_(i)), i=1, 2, . . . , N. Initially all the weights are set tow_(i)=1/N, so that the first step simply trains the decision rule on thedata in the usual manner. For each successive iteration m=2, 3, . . . ,M the observation weights are individually modified and the decisionrule is reapplied to the weighted observations. At step m, thoseobservations that were misclassified by the decision rule G_(m)−1 (x)induced at the previous step have their weights increased, whereas theweights are decreased for those that were classified correctly. Thus asiterations proceed, observations that are difficult to correctlyclassify receive ever-increasing influence. Each successive decisionrule is thereby forced to concentrate on those training observationsthat are missed by previous ones in the sequence.

The exemplary boosting algorithm is summarized as follows:

-   -   1. Initialize the observation weights w_(i)=1/N, i=1, 2, . . . ,        N.    -   2. For m=1 to M:        -   (a) Fit a decision rule G_(m)(x) to the training set using            weights w_(i).        -   (b) Compute

${err}_{m} = \frac{\sum\limits_{i = 1}^{N}\;{w_{i}{I\left( {y_{i} \neq {G_{m}\left( x_{i} \right)}} \right)}}}{\sum\limits_{i = 1}^{N}\; w_{i}}$

-   -   -   (c) Compute α_(m)=log((1−err_(m))/err_(m)).        -   (d) Set w_(i)←w_(i)·exp [α_(m)·I(y_(i)≠G_(m)(x_(i)))], i=1,            2, . . . , N.

    -   3. Output G(x)=sign └Σ_(m=1) ^(M)α_(m)G_(m)(x)┘

In one embodiment in accordance with this algorithm, each object is, infact, an observable. Furthermore, in the algorithm, the current decisionrule G_(m)(x) is induced on the weighted observations at line 2a. Theresulting weighted error rate is computed at line 2b. Line 2c calculatesthe weight α_(m) given to G_(m)(x) in producing the final classifierG(x) (line 3). The individual weights of each of the observations areupdated for the next iteration at line 2d. Observations misclassified byG_(m)(x) have their weights scaled by a factor exp(a_(m)), increasingtheir relative influence for inducing the next classifier G_(m)+1(x) inthe sequence. In some embodiments, modifications are used of theboosting methods in Freund and Schapire, 1997, Journal of Computer andSystem Sciences 55, pp. 119-139, the entire contents of which are herebyincorporated by reference herein. See, for example, Hasti et al., TheElements of Statistical Learning, 2001, Springer, New York, Chapter 10,the entire contents of which are hereby incorporated by referenceherein.

For example, in some embodiments, observable preselection is performedusing a technique such as the nonparametric scoring methods of Park etal., 2002, Pac. Symp. Biocomput. 6, 52-63, the entire contents of whichare hereby incorporated by reference herein. Observable preselection isa form of dimensionality reduction in which the observables thatdiscriminate between phenotypic classifications the best are selectedfor use in the classifier. Examples of observables include, but are notlimited to, values of specific pixels, patterns of values of specificgroups of pixels, values of specific measured wavelengths or any otherform of observable data that is directly present in the spectral dataand/or that can be derived from the spectral data. Next, the LogitBoostprocedure introduced by Friedman et al., 2000, Ann Stat 28, 337-407, theentire contents of which are hereby incorporated by reference herein, isused rather than the boosting procedure of Freund and Schapire. In someembodiments, the boosting and other classification methods of Ben-Dor etal., 2000, Journal of Computational Biology 7, 559-583, herebyincorporated by reference in its entirety, are used. In someembodiments, the boosting and other classification methods of Freund andSchapire, 1997, Journal of Computer and System Sciences 55, 119-139, theentire contents of which are hereby incorporated by reference herein,are used.

In the random subspace method, decision rules are constructed in randomsubspaces of the data feature space. These decision rules are usuallycombined by simple majority voting in the final decision rule. See, forexample, Ho, “The Random subspace method for constructing decisionforests,” IEEE Trans Pattern Analysis and Machine Intelligence, 1998;20(8): 832-844, the entire contents of which are incorporated byreference herein.

IV. Multiple Additive Regression Trees

Multiple additive regression trees (MART) represent another way toconstruct a decision rule. A generic algorithm for MART is:

-   -   1. Initialize f₀(x)=arg min γΣ_(i=1) ^(N)L(y_(i),γ).    -   2. For m=1 to M:        -   (a) For i=1, 2, . . . , N compute

$r_{im} = {- \left\lbrack \frac{\partial{L\left( {y_{i},{f\left( x_{i} \right)}} \right)}}{\partial{f\left( x_{i} \right)}} \right\rbrack_{f = f_{m - 1}}}$

-   -   -   (b) Fit a regression tree to the targets r_(im) giving            terminal regions R_(jm), j=1, 2, . . . , J_(m).        -   (c) For j=1, 2, . . . , J_(m) compute

$\gamma_{jm} = {\arg{\min\limits_{\gamma}{\sum\limits_{x_{i} \in R_{jm}}\;{{L\left( {y_{i},{{f_{m - 1}\left( x_{i} \right)} + \gamma}} \right)}.}}}}$

-   -   -   (d) Update f_(m)(x)=f_(m-1)(x)+Σ_(j=1) ^(J) ^(m)            γ_(jm)I(x∈R_(jm))

    -   3. Output {circumflex over (f)}(x)=f_(M)(x).

    -   Specific algorithms are obtained by inserting different loss        criteria L(y,f(x)). The first line of the algorithm initializes        to the optimal constant model, which is just a single terminal        node tree. The components of the negative gradient computed in        line 2(a) are referred to as generalized pseudo residuals, r.        Gradients for commonly used loss functions are summarized in        Table 10.2, of Hastie et al., 2001, The Elements of Statistical        Learning, Springer-Verlag, New York, p. 321, the entire contents        of which are hereby incorporated by reference herein. The        algorithm for classification is similar and is described in        Hastie et al., Chapter 10, the entire contents of which are        hereby incorporated by reference herein. Tuning parameters        associated with the MART procedure are the number of iterations        M and the sizes of each of the constituent trees J_(m), m=1, 2,        . . . , M.

V. Decision Rules Derived by Regression

In some embodiments, a decision rule used to classify subjects is builtusing regression. In such embodiments, the decision rule can becharacterized as a regression classifier, such as a logistic regressionclassifier. Such a regression classifier includes a coefficient for aplurality of observables from the spectral training data that is used toconstruct the classifier. Examples of such observables in the spectraltraining set include, but are not limited to values of specific pixels,patterns of values of specific groups of pixels, values of specificmeasured wavelengths or any other form of observable data that isdirectly present in the spectral data and/or that can be derived fromthe spectral data. In such embodiments, the coefficients for theregression classifier are computed using, for example, a maximumlikelihood approach.

In one specific embodiment, the training population includes a pluralityof trait subgroups (e.g., three or more trait subgroups, four or morespecific trait subgroups, etc.). These multiple trait subgroups cancorrespond to discrete stages of a biological insult such as a lesion.In this specific embodiment, a generalization of the logistic regressionmodel that handles multicategory responses can be used to develop adecision that discriminates between the various trait subgroups found inthe training population. For example, measured data for selectedobservables can be applied to any of the multi-category logit modelsdescribed in Agresti, An Introduction to Categorical Data Analysis,1996, John Wiley & Sons, Inc., New York, Chapter 8, the entire contentsof which are hereby incorporated by reference herein, in order todevelop a classifier capable of discriminating between any of aplurality of trait subgroups represented in a training population.

VI. Neural Networks

In some embodiments, spectral data training sets can be used to train aneural network. A neural network is a two-stage regression orclassification decision rule. A neural network has a layered structurethat includes a layer of input units (and the bias) connected by a layerof weights to a layer of output units. For regression, the layer ofoutput units typically includes just one output unit. However, neuralnetworks can handle multiple quantitative responses in a seamlessfashion.

In multilayer neural networks, there are input units (input layer),hidden units (hidden layer), and output units (output layer). There is,furthermore, a single bias unit that is connected to each unit otherthan the input units. Neural networks are described in Duda et al.,2001, Pattern Classification, Second Edition, John Wiley & Sons, Inc.,New York; and Hastie et al., 2001, The Elements of Statistical Learning,Springer-Verlag, New York, the entire contents of each of which arehereby incorporated by reference herein. Neural networks are alsodescribed in Draghici, 2003, Data Analysis Tools for DNA Microarrays,Chapman & Hall/CRC; and Mount, 2001, Bioinformatics: sequence and genomeanalysis, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, N.Y.,the entire contents of each of which are incorporated by referenceherein. What are disclosed below is some exemplary forms of neuralnetworks.

One basic approach to the use of neural networks is to start with anuntrained network, present a training pattern to the input layer, and topass signals through the net and determine the output at the outputlayer. These outputs are then compared to the target values; anydifference corresponds to an error. This error or criterion function issome scalar function of the weights and is minimized when the networkoutputs match the desired outputs. Thus, the weights are adjusted toreduce this measure of error. For regression, this error can besum-of-squared errors. For classification, this error can be eithersquared error or cross-entropy (deviation). See, e.g., Hastie et al.,2001, The Elements of Statistical Learning, Springer-Verlag, New York,the entire contents of which are hereby incorporated by referenceherein.

Three commonly used training protocols are stochastic, batch, andon-line. In stochastic training, patterns are chosen randomly from thetraining set and the network weights are updated for each patternpresentation. Multilayer nonlinear networks trained by gradient descentmethods such as stochastic back-propagation perform a maximum-likelihoodestimation of the weight values in the classifier defined by the networktopology. In batch training, all patterns are presented to the networkbefore learning takes place. Typically, in batch training, severalpasses are made through the training data. In online training, eachpattern is presented once and only once to the net.

In some embodiments, consideration is given to starting values forweights. If the weights are near zero, then the operative part of thesigmoid commonly used in the hidden layer of a neural network (see,e.g., Hastie et al., 2001, The Elements of Statistical Learning,Springer-Verlag, New York, the entire contents of which are herebyincorporated by reference herein) is roughly linear, and hence theneural network collapses into an approximately linear classifier. Insome embodiments, starting values for weights are chosen to be randomvalues near zero. Hence the classifier starts out nearly linear, andbecomes nonlinear as the weights increase. Individual units localize todirections and introduce nonlinearities where needed. Use of exact zeroweights leads to zero derivatives and perfect symmetry, and thealgorithm never moves. Alternatively, starting with large weights oftenleads to poor solutions.

Since the scaling of inputs determines the effective scaling of weightsin the bottom layer, it can have a large effect on the quality of thefinal solution. Thus, in some embodiments, at the outset all expressionvalues are standardized to have mean zero and a standard deviation ofone. This ensures all inputs are treated equally in the regularizationprocess, and allows one to choose a meaningful range for the randomstarting weights. With standardization inputs, it is typical to takerandom uniform weights over the range [−0.7, +0.7].

A recurrent problem in the use of three-layer networks is the optimalnumber of hidden units to use in the network. The number of inputs andoutputs of a three-layer network are determined by the problem to besolved. In the present application, the number of inputs for a givenneural network will equal the number of observables selected from thetraining population. Here, an observable can be, for example, measuredvalues for specific pixels in an image, measured values for specificwavelengths in an image, where the image is from a single spectralsource or from a fusion of two or more disparate spectral sources. Thenumber of outputs for the neural network will typically be just one.However, in some embodiments, more than one output is used so that morethan just two states can be defined by the network. For example, amulti-output neural network can be used to discriminate between healthyphenotypes, sick phenotypes, and various stages in between. If too manyhidden units are used in a neural network, the network will have toomany degrees of freedom and is trained too long, there is a danger thatthe network will overfit the data. If there are too few hidden units,the training set cannot be learned. Generally speaking, however, it isbetter to have too many hidden units than too few. With too few hiddenunits, the classifier might not have enough flexibility to capture thenonlinearities in the date; with too many hidden units, the extra weightcan be shrunk towards zero if appropriate regularization or pruning, asdescribed below, is used. In typical embodiments, the number of hiddenunits is somewhere in the range of 5 to 100, with the number increasingwith the number of inputs and number of training cases.

One general approach to determining the number of hidden units to use isto apply a regularization approach. In the regularization approach, anew criterion function is constructed that depends not only on theclassical training error, but also on classifier complexity.Specifically, the new criterion function penalizes highly complexclassifiers; searching for the minimum in this criterion is to balanceerror on the training set with error on the training set plus aregularization term, which expresses constraints or desirable propertiesof solutions:J=J _(pat) +λJ _(reg).The parameter λ is adjusted to impose the regularization more or lessstrongly. In other words, larger values for λ will tend to shrinkweights towards zero: typically cross-validation with a validation setis used to estimate λ. This validation set can be obtained by settingaside a random subset of the training population. Other forms of penaltyhave been proposed, for example the weight elimination penalty (see,e.g., Hastie et al., 2001, The Elements of Statistical Learning,Springer-Verlag, New York, the entire contents of which are incorporatedby reference herein).

Another approach to determine the number of hidden units to use is toeliminate—prune—weights that are least needed. In one approach, theweights with the smallest magnitude are eliminated (set to zero). Suchmagnitude-based pruning can work, but is nonoptimal; sometimes weightswith small magnitudes are important for learning and training data. Insome embodiments, rather than using a magnitude-based pruning approach,Wald statistics are computed. The fundamental idea in Wald Statistics isthat they can be used to estimate the importance of a hidden unit(weight) in a classifier. Then, hidden units having the least importanceare eliminated (by setting their input and output weights to zero). Twoalgorithms in this regard are the Optimal Brain Damage (OBD) and theOptimal Brain Surgeon (OBS) algorithms that use second-orderapproximation to predict how the training error depends upon a weight,and eliminate the weight that leads to the smallest increase in trainingerror.

Optimal Brain Damage and Optimal Brain Surgeon share the same basicapproach of training a network to local minimum error at weight w, andthen pruning a weight that leads to the smallest increase in thetraining error. The predicted functional increase in the error for achange in full weight vector δw is:

${\delta\; J} = {{{\left( \frac{\partial J}{\partial w} \right)^{t} \cdot \delta}\; w} + {\frac{1}{2}\delta\;{w^{t} \cdot \frac{\partial^{2}J}{\partial w^{2}} \cdot \delta}\; w} + {O\left( {{\delta\; w}}^{3} \right)}}$where

$\frac{\partial^{2}J}{\partial w^{2}}$is the Hessian matrix. The first term vanishes at a local minimum inerror; third and higher order terms are ignored. The general solutionfor minimizing this function given the constraint of deleting one weightis:

${\delta\; w} = {{{- \frac{w_{q}}{\left\lbrack H^{- 1} \right\rbrack_{qq}}}{H^{- 1} \cdot u_{q}}\mspace{14mu}{and}\mspace{14mu} L_{q}} = {\frac{1}{2} - \frac{w_{q}^{2}}{\left\lbrack H^{- 1} \right\rbrack_{qq}}}}$Here, u_(q) is the unit vector along the qth direction in weight spaceand L_(q) is approximation to the saliency of the weight q−the increasein training error if weight q is pruned and the other weights updatedδw. These equations require the inverse of H. One method to calculatethis inverse matrix is to start with a small value, H₀ ⁻¹=α⁻¹I, where αis a small parameter—effectively a weight constant. Next the matrix isupdated with each pattern according to

$\begin{matrix}{H_{m + 1}^{- 1} = {H_{m}^{- 1} - \frac{H_{m}^{- 1}X_{m + 1}X_{m + 1}^{T}H_{m}^{- 1}}{\frac{n}{a_{m}} + {X_{m + 1}^{T}H_{m}^{- 1}X_{m + 1}}}}} & \left( {{Eqn}.\mspace{14mu} 1} \right)\end{matrix}$where the subscripts correspond to the pattern being presented and a_(m)decreases with m. After the full training set has been presented, theinverse Hessian matrix is given by H⁻¹=H_(n) ⁻¹. In algorithmic form,the Optimal Brain Surgeon method is:

  begin initialize n_(H), w, θ  train a reasonably large network tominimum error  do compute H⁻¹ by Eqn. 1    $\left. {q\text{*}}\leftarrow{\arg\mspace{14mu}{\min\limits_{q}\mspace{11mu}{{w_{q}^{2}/\left( {2\left\lbrack H^{- 1} \right\rbrack}_{qq} \right)}\mspace{14mu}\left( {{saliency}\mspace{14mu} L_{q}} \right)}}} \right.$   $\left. w\leftarrow{w - {\frac{w_{q\text{*}}}{\left\lbrack H^{- 1} \right\rbrack_{q\text{*}q\text{*}}}H^{- 1}e_{q\text{*}}\mspace{14mu}\left( {{saliency}\mspace{14mu} L_{q}} \right)}} \right.$  until J(w) > θ  return w end

The Optimal Brain Damage method is computationally simpler because thecalculation of the inverse Hessian matrix in line 3 is particularlysimple for a diagonal matrix. The above algorithm terminates when theerror is greater than a criterion initialized to be θ. Another approachis to change line 6 to terminate when the change in J(w) due toelimination of a weight is greater than some criterion value. In someembodiments, the back-propagation neural network. See, for example Abdi,1994, “A neural network primer,” J. Biol System. 2, 247-283, the entirecontents of which are incorporated by reference herein.

VII. Clustering

In some embodiments, observables in the spectral data sets such asvalues of specific pixels, patterns of values of specific groups ofpixels, values of specific measured wavelengths or any other form ofobservable data that is directly present in the data or that can bederived from the data are used to cluster a training set. For example,consider the case in which ten such observables are used. Each member mof the training population will have values for each of the tenobservable. Such values from a member m in the training populationdefine the vector:

-   -   X_(1m) X_(2m) X_(3m) X_(4m) X_(5m) X_(6m) X_(7m) X_(8m) X_(9m)        X_(10m)        where X_(im) is the measured or derived value of the i^(th)        observable in a spectral data set m. If there are m spectral        data sets in the training set, where each such data set        corresponds to a subject having known phenotypic classification        or each such data set corresponds to the same subject having        known phenotypic classification but at a unique time point,        selection of i observables will define m vectors. Note that        there is no requirement that the measured or derived value of        every single observable used in the vectors be represented in        every single vector m. In other words, spectral data from a        subject in which one of the i^(th) observables is not found can        still be used for clustering. In such instances, the missing        observable is assigned either a “zero” or some other value. In        some embodiments, prior to clustering, the values for the        observables are normalized to have a mean value of zero and unit        variance.

Those members of the training population that exhibit similar values forcorresponding observables will tend to cluster together. A particularcombination of observables is considered to be a good classifier whenthe vectors cluster into the trait groups found in the trainingpopulation. For instance, if the training population includes class a:subjects that do not have the medical condition, and class b: subjectsthat do have the medical condition, a useful clustering classifier willcluster the population into two groups, with one cluster group uniquelyrepresenting class a and the other cluster group uniquely representingclass b.

Clustering is described on pages 211-256 of Duda and Hart, PatternClassification and Scene Analysis, 1973, John Wiley & Sons, Inc., NewYork, (hereinafter “Duda 1973”) which is hereby incorporated byreference in its entirety. As described in Section 6.7 of Duda 1973, theclustering problem is described as one of finding natural groupings in adataset. To identify natural groupings, two issues are addressed. First,a way to measure similarity (or dissimilarity) between two samples isdetermined. This metric (similarity measure) is used to ensure that thesamples in one cluster are more like one another than they are tosamples in other clusters. Second, a mechanism for partitioning the datainto clusters using the similarity measure is determined.

Similarity measures are discussed in Section 6.7 of Duda 1973, where itis stated that one way to begin a clustering investigation is to definea distance function and to compute the matrix of distances between allpairs of samples in a dataset. If distance is a good measure ofsimilarity, then the distance between samples in the same cluster willbe significantly less than the distance between samples in differentclusters. However, as stated on page 215 of Duda 1973, clustering doesnot require the use of a distance metric. For example, a nonmetricsimilarity function s(x, x′) can be used to compare two vectors x andx′. Conventionally, s(x, x′) is a symmetric function whose value islarge when x and x′ are somehow “similar”. An example of a nonmetricsimilarity function s(x, x′) is provided on page 216 of Duda 1973.

Once a method for measuring “similarity” or “dissimilarity” betweenpoints in a dataset has been selected, clustering requires a criterionfunction that measures the clustering quality of any partition of thedata. Partitions of the data set that extremize the criterion functionare used to cluster the data. See page 217 of Duda 1973. Criterionfunctions are discussed in Section 6.8 of Duda 1973.

More recently, Duda et al., Pattern Classification, 2^(nd) edition, JohnWiley & Sons, Inc. New York, has been published. Pages 537-563 provideadditional clustering details. More information on clustering techniquescan be found in the following references, the entire contents of each ofwhich are hereby incorporated by reference herein: Kaufman andRousseeuw, 1990, Finding Groups in Data: An Introduction to ClusterAnalysis, Wiley, New York, N.Y.; Everitt, 1993, Cluster analysis (3ded.), Wiley, New York, N.Y.; and Backer, 1995, Computer-AssistedReasoning in Cluster Analysis, Prentice Hall, Upper Saddle River, N.J.Particular exemplary clustering techniques that can be used include, butare not limited to, hierarchical clustering (agglomerative clusteringusing nearest-neighbor algorithm, farthest-neighbor algorithm, theaverage linkage algorithm, the centroid algorithm, or the sum-of-squaresalgorithm), k-means clustering, fuzzy k-means clustering algorithm, andJarvis-Patrick clustering.

VIII. Principle Component Analysis

Principal component analysis (PCA) can be used to analyze observables inthe spectral data sets such as values of specific pixels, patterns ofvalues of specific groups of pixels, values of specific measuredwavelengths or any other form of observable data that is directlypresent in the spectral data or that can be derived from the spectraldata in order to construct a decision rule that discriminates subjectsin the training set. Principal component analysis is a classicaltechnique to reduce the dimensionality of a data set by transforming thedata to a new set of variable (principal components) that summarize thefeatures of the data. See, for example, Jolliffe, 1986, PrincipalComponent Analysis, Springer, New York, which is hereby incorporated byreference in its entirety. Principal component analysis is alsodescribed in Draghici, 2003, Data Analysis Tools for DNA Microarrays,Chapman & Hall/CRC, which is hereby incorporated by reference in itsentirety. What follows are some non-limiting examples of principalcomponents analysis.

Principal components (PCs) are uncorrelated and are ordered such thatthe k^(th) PC has the kth largest variance among PCs. The k^(th) PC canbe interpreted as the direction that maximizes the variation of theprojections of the data points such that it is orthogonal to the firstk−1 PCs. The first few PCs capture most of the variation in the dataset. In contrast, the last few PCs are often assumed to capture only theresidual ‘noise’ in the data.

PCA can also be used to create a classifier. In such an approach,vectors for selected observables can be constructed in the same mannerdescribed for clustering above. The set of vectors, where each vectorrepresents the measured or derived values for the select observablesfrom a particular member of the training population, can be viewed as amatrix. In some embodiments, this matrix is represented in a Free-Wilsonmethod of qualitative binary description of monomers (Kubinyi, 1990, 3DQSAR in drug design theory methods and applications, Pergamon Press,Oxford, pp 589-638), and distributed in a maximally compressed spaceusing PCA so that the first principal component (PC) captures thelargest amount of variance information possible, the second principalcomponent (PC) captures the second largest amount of all varianceinformation, and so forth until all variance information in the matrixhas been considered.

Then, each of the vectors (where each vector represents a member of thetraining population, or each vector represents a member of the trainingpopulation at a specific instance in time) is plotted. Many differenttypes of plots are possible. In some embodiments, a one-dimensional plotis made. In this one-dimensional plot, the value for the first principalcomponent from each of the members of the training population isplotted. In this form of plot, the expectation is that members of afirst subgroup (e.g. those subjects that have a first type of lesion)will cluster in one range of first principal component values andmembers of a second subgroup (e.g., those subjects that have a secondtype of lesion) will cluster in a second range of first principalcomponent values.

In one example, the training population includes two subgroups: “haslesion” and “does not have lesion.” The first principal component iscomputed using the values of observables across the entire trainingpopulation data set. Then, each member of the training set is plotted asa function of the value for the first principal component. In thisexample, those members of the training population in which the firstprincipal component is positive are classified as “has lesion” and thosemembers of the training population in which the first principalcomponent is negative are classified as “does not have lesion.”

In some embodiments, the members of the training population are plottedagainst more than one principal component. For example, in someembodiments, the members of the training population are plotted on atwo-dimensional plot in which the first dimension is the first principalcomponent and the second dimension is the second principal component. Insuch a two-dimensional plot, the expectation is that members of eachsubgroup represented in the training population will cluster intodiscrete groups. For example, a first cluster of members in thetwo-dimensional plot will represent subjects that have a first type oflesion and a second cluster of members in the two-dimensional plot willrepresent subjects that have a second type of lesion.

IX. Nearest Neighbor Analysis

Nearest neighbor classifiers are memory-based and require no classifierto be fit. Given a query point x₀, the k training points x_((r)), r, . .. , k closest in distance to x₀ are identified and then the point x₀ isclassified using the k nearest neighbors. Ties can be broken at random.In some embodiments, Euclidean distance in feature space is used todetermine distance as:d _((i)) =∥x _((i)) −x _(o)∥.In some embodiments, when the nearest neighbor algorithm is used, theobservables in the spectral data used to compute the linear discriminantis standardized to have mean zero and variance 1.

The members of the training population can be randomly divided into atraining set and a test set. For example, in one embodiment, two thirdsof the members of the training population are placed in the training setand one third of the members of the training population are placed inthe test set. A select combination of observables represents the featurespace into which members of the test set are plotted. Observables in thespectral data include, but are not limited to values of specific pixels,patterns of values of specific groups of pixels, values of specificmeasured wavelengths or any other form of observable data that isdirectly present in the spectral data and/or that can be derived fromthe spectral data.

Next, the ability of the training set to correctly characterize themembers of the test set is computed. In some embodiments, nearestneighbor computation is performed several times for a given combinationof spectral features. In each iteration of the computation, the membersof the training population are randomly assigned to the training set andthe test set. Then, the quality of the combination of observables chosento develop the classifier is taken as the average of each such iterationof the nearest neighbor computation.

The nearest neighbor rule can be refined to deal with issues of unequalclass priors, differential misclassification costs, and featureselection. Many of these refinements involve some form of weightedvoting for the neighbors. For more information on nearest neighboranalysis, see Duda, Pattern Classification, Second Edition, 2001, JohnWiley & Sons, Inc; and Hastie, 2001, The Elements of StatisticalLearning, Springer, New York, each of which is hereby incorporated byreference in its entirety.

X. Linear Discriminant Analysis

Linear discriminant analysis (LDA) attempts to classify a subject intoone of two categories based on certain object properties. In otherwords, LDA tests whether object attributes measured in an experimentpredict categorization of the objects. LDA typically requires continuousindependent variables and a dichotomous categorical dependent variable.The feature values for selected combinations of observables across asubset of the training population serve as the requisite continuousindependent variables. The trait subgroup classification of each of themembers of the training population serves as the dichotomous categoricaldependent variable. LDA seeks the linear combination of variables thatmaximizes the ratio of between-group variance and within-group varianceby using the grouping information. Implicitly, the linear weights usedby LDA depend on how the measured values of an observable across thetraining set separates in the two groups (e.g., a group a that haslesion type 1 and a group b that has lesion type b) and how thesemeasured values correlate with the measured values of other observables.In some embodiments, LDA is applied to the data matrix of the N membersin the training sample by K observables in a combination of observables.Observables in the spectral data sets are, for example, values ofspecific pixels, patterns of values of specific groups of pixels, valuesof specific measured wavelengths or any other form of observable datathat is directly present in the spectral data and/or that can be derivedfrom the spectral data. Then, the linear discriminant of each member ofthe training population is plotted. Ideally, those members of thetraining population representing a first subgroup (e.g. “sick” subjects)will cluster into one range of linear discriminant values (e.g.,negative) and those member of the training population representing asecond subgroup (e.g. “healthy” subjects) will cluster into a secondrange of linear discriminant values (e.g., positive). The LDA isconsidered more successful when the separation between the clusters ofdiscriminant values is larger. For more information on lineardiscriminant analysis, see Duda, Pattern Classification, Second Edition,2001, John Wiley & Sons, Inc; and Hastie, 2001, The Elements ofStatistical Learning, Springer, New York; and Venables & Ripley, 1997,Modern Applied Statistics with s-plus, Springer, New York, each of whichis hereby incorporated by reference in its entirety.

XI. Quadratic Discriminant Analysis

Quadratic discriminant analysis (QDA) takes the same input parametersand returns the same results as LDA. QDA uses quadratic equations,rather than linear equations, to produce results. LDA and QDA areinterchangeable, and which to use is a matter of preference and/oravailability of software to support the analysis. Logistic regressiontakes the same input parameters and returns the same results as LDA andQDA.

XII. Support Vector Machines

In some embodiments, support vector machines (SVMs) are used to classifysubjects using values of specific predetermined observables. Observablesin the training data, include, but are not limited to values of specificpixels, patterns of values of specific groups of pixels, values ofspecific measured wavelengths or any other form of observable data thatis directly present in the spectral data and/or that can be derived fromthe spectral data. SVMs are a relatively new type of learning algorithm.See, for example, Cristianini and Shawe-Taylor, 2000, An Introduction toSupport Vector Machines, Cambridge University Press, Cambridge; Boser etal., 1992, “A training algorithm for optimal margin classifiers,” inProceedings of the 5^(th) Annual ACM Workshop on Computational LearningTheory, ACM Press, Pittsburgh, Pa., pp. 142-152; Vapnik, 1998,Statistical Learning Theory, Wiley, New York; Mount, 2001,Bioinformatics: sequence and genome analysis, Cold Spring HarborLaboratory Press, Cold Spring Harbor, N.Y., Duda, PatternClassification, Second Edition, 2001, John Wiley & Sons, Inc.; andHastie, 2001, The Elements of Statistical Learning, Springer, New York;and Furey et al., 2000, Bioinformatics 16, 906-914, each of which ishereby incorporated by reference in its entirety.

When used for classification, SVMs separate a given set of binarylabeled data training data with a hyper-plane that is maximallydistanced from them. For cases in which no linear separation ispossible, SVMs can work in combination with the technique of ‘kernels’,which automatically realizes a non-linear mapping to a feature space.The hyper-plane found by the SVM in feature space corresponds to anon-linear decision boundary in the input space.

In one approach, when a SVM is used, the feature data is standardized tohave mean zero and unit variance and the members of a trainingpopulation are randomly divided into a training set and a test set. Forexample, in one embodiment, two thirds of the members of the trainingpopulation are placed in the training set and one third of the membersof the training population are placed in the test set. The observedvalues for a combination of observables in the training set are used totrain the SVM. Then the ability for the trained SVM to correctlyclassify members in the test set is determined. In some embodiments,this computation is performed several times for a given combination ofspectral features. In each iteration of the computation, the members ofthe training population are randomly assigned to the training set andthe test set. Then, the quality of the combination of observables istaken as the average of each such iteration of the SVM computation.

XIII. Evolutionary Methods

Inspired by the process of biological evolution, evolutionary methods ofdecision rule design employ a stochastic search for a decision rule. Inbroad overview, such methods create several decision rules—apopulation—from a combination of observables in the training set.Observables in the training set are, for example, values of specificpixels, patterns of values of specific groups of pixels, values ofspecific measured wavelengths or any other form of observable data thatis directly present in the spectral data and/or that can be derived fromthe spectral data. Each decision rule varies somewhat from the other.Next, the decision rules are scored on observable measured across thetraining population. In keeping with the analogy with biologicalevolution, the resulting (scalar) score is sometimes called the fitness.The decision rules are ranked according to their score and the bestdecision rules are retained (some portion of the total population ofdecision rules). Again, in keeping with biological terminology, this iscalled survival of the fittest. The decision rules are stochasticallyaltered in the next generation—the children or offspring. Some offspringdecision rules will have higher scores than their parent in the previousgeneration, some will have lower scores. The overall process is thenrepeated for the subsequent generation: the decision rules are scoredand the best ones are retained, randomly altered to give yet anothergeneration, and so on. In part, because of the ranking, each generationhas, on average, a slightly higher score than the previous one. Theprocess is halted when the single best decision rule in a generation hasa score that exceeds a desired criterion value. More information onevolutionary methods is found in, for example, Duda, PatternClassification, Second Edition, 2001, John Wiley & Sons, Inc, which ishereby incorporated by reference herein in its entirety.

XIV. Additional Embodiments

Additional methods for processing spectral data for use in tissuecharacterization may be applied either at the hyperspectral image stage102 or at the contact probe stage 107. For example, a subset of data maybe identified among the hyperspectral data from the plurality of regionsmeasured in step 102. The subset of data may be identified because it isaffected by glare, shadow, mucus, a smoke tube, and/or speculum. Theregions of the subject's skin that correspond to the subset of data willalso be identified. The affected subset of data will either bedisregarded in future processing, or weighted for partial considerationin future processing, or fully processed like the rest of the data set.In some embodiments, the regions of skin corresponding to the affectedsubset of data are measured in a repeat instance of step 102 after theobstruction has been removed or treated. In some embodiments, the regionof skin corresponding to the affected subset of data is measured in aninstance of step 107 and/or one more repeat instances of step 107 ofFIG. 1A after the obstruction has been removed or treated.

In other embodiments, the subset set of data may be identified in datacollected in each region (including the suspect region) and used ascontrol in regression analysis. In one example, a randomly selectedsubset set of data is not subjected to data modification and is used asan internal standard to calibrate and control the overall quality ofdata processing. It is also possible to spite the same region of asubject's skin into subregions and process the subsets by differentmethods. Results from the individual subsets can then be combined ormodified in order to determine an overall result, for example, an imagethat corresponds to the entire region. The data splitting may also beperformed randomly or based on sub-regional locations.

Additional exemplary methods for data processing include but are notlimited to, for example, U.S. Pat. No. 7,282,723 to Schomacker issued onOct. 16, 2007 (Methods and Apparatus for Processing Spectral data foruse in Tissue Characterization); U.S. patent application to Garini etal., published as Publication No. 2003/0215791 A1 (Method of and Systemfor Multiplexed Analysis by Spectral Imaging); and U.S. Pat. No.7,219,086 B2 to Geshwind et al. issued on May 15, 2007 (System andMethod for Hyper-Spectral Analysis); each of which is herebyincorporated by reference herein in its entirety.

The aforementioned methods for tissue characterization may be usedindividually or in a combination with each other.

D. Combining Decision Rules to Classify a Subject

In some embodiments, multiple decision rules are used to identify afeature of biological interest in a subject's skin (e.g., a lesion), tocharacterize such a feature (e.g., to identify a type of skin lesion),or to detect a change in a skin lesion over time. For instance, a firstdecision rule may be used to determine whether a subject has a skinlesion and, if the subject does have a skin lesion, a second decisionrule may be used to determine whether a subject has a specific type ofskin lesion. Advantageously, and as described above, in some instancessuch decision rules can be trained using a training data set thatincludes hyperspectral imaging data from subjects with known phenotype(e.g., lesions of known type). As such, in some embodiments of thepresent disclosure, a particular decision rule is not executed unlessmodel preconditions associated with the decision rule have beensatisfied.

For example, in some embodiments, a model precondition specifies that afirst decision rule that is indicative of a broader biological sampleclass (e.g., a more general phenotype) than a second decision rule mustbe run before the second decision rule, indicative of a narrowerbiological sample class, is run. To illustrate, a model precondition ofa second decision rule that is indicative of a particular form of skinlesion could require that a first decision rule, that is indicative ofskin lesion generally, test positive prior to running the seconddecision rule. In some embodiments, a model precondition includes arequirement that another decision rule in a plurality of decision rulesbe identified as negative, positive, or indeterminate prior to testinganother decision rule. A few additional examples of how preconditionscan be used to arrange decision rules into hierarchies follow.

In a first example, the preconditions of decision rule B require thatdecision rule A have a specific result before decision rule B is run. Itmay well be the case that decision rule A is run, yet fails to yield thespecific result required by decision rule B. In this case, decision ruleB is never run. If, however, decision rule A is run and yields thespecific result required by decision rule B, then decision rule B isrun. This example can be denoted as:

-   -   if (A=result), then B can be run.

In a second example, the preconditions of decision rule C require thateither decision rule A has a specific result or that decision rule B hasa specific result prior to running decision rule C. This example can bedenoted as:

-   -   if ((A=first result) or (B=second result)), then C can be run.

To illustrate, a model C can require that decision rule A be run andtest positive for a skin lesion type A or that decision rule B be runand test positive for skin lesion type B, before decision rule C is run.Alternatively, the preconditions of decision rule C could require thatboth decision rule A and decision rule B achieve specific results:

-   -   if ((A=first result) and (B=second result)), then C can be run.

In another example, the preconditions of decision rule D require thatdecision rule C has a specific result before decision rule D is run. Thepreconditions of decision rule C, in turn, require that decision rule Ahas a first result and that decision rule B has a second result beforedecision rule C is run. This example can be denoted as:

-   -   If ((A=first result) and (B=second result)), then C can be run    -   If (C=third result), then D can be run.

These examples illustrate the advantages that model preconditionsprovide. Because of the preconditions of the present application,decision rules can be arranged into hierarchies in which specificdecision rules are run before other decision rules are run. Often, thedecision rules run first are designed to classify a subject into a broadbiological sample class (e.g., broad phenotype). Once the subject hasbeen broadly classified, subsequent decision rules are run to refine thepreliminary classification into a narrower biological sample class(e.g., a specific skin lesion type or state).

E. Sharing Hyperspectral Images with Third Parties

Because hyperspectral data cubes and the raw output of other types ofsensors/cameras can contain a tremendous amount of information, sharingsuch data with third parties can be impeded by finite transfer ratesand/or finite storage space. However, because not all of the informationin hyperspectral data cubes and/or raw sensor output is useful incharacterizing a medical condition, the medical information within thatdata can usefully be shared with third parties in the form of “outline”or “shape” files that can be overlaid against conventional images of thesubject. The “outline” files can indicate the location and boundary ofthe medical condition, and can include a description of the medicalcondition. In some embodiments, the “outline” files include an intensitymap generated by the image constructor described above. A frame ofreference for the file (e.g., the location on the subject's body towhich the file corresponds) can also be transmitted to the third party.

4. Other Embodiments

The systems and methods described herein can be used to determinewhether the subject has a wide variety of medical conditions. Someexamples include, but are not limited to: abrasion, alopecia, atrophy,av malformation, battle sign, bullae, burrow, basal cell carcinoma,burn, candidal diaper dermatitis, cat-scratch disease, contactdermatitis, cutaneous larva migrans, cutis marmorata, dermatoma,ecchymosis, ephelides, erythema infectiosum, erythema multiforme,eschar, excoriation, fifth disease, folliculitis, graft vs. hostdisease, guttate, guttate psoriasis, hand, foot and mouth disease,Henoch-Schonlein purpura, herpes simplex, hives, id reaction, impetigo,insect bite, juvenile rheumatoid arthritis, Kawasaki disease, keloids,keratosis pilaris, Koebner phenomenon, Langerhans cell histiocytosis,leukemia, lichen striatus, lichenification, livedo reticularis,lymphangitis, measles, meningococcemia, molluscum contagiosum,neurofibromatosis, nevus, poison ivy dermatitis, psoriasis, scabies,scarlet fever, scar, seborrheic dermatitis, serum sickness, Shagreenplaque, Stevens-Johnson syndrome, strawberry tongue, swimmers' itch,telangiectasia, tinea capitis, tinea corporis, tuberous sclerosis,urticaria, varicella, varicella zoster, wheal, xanthoma, zosteriform,basal cell carcinoma, squamous cell carcinoma, malignant melanoma,dermatofibrosarcoma protuberans, Merkel cell carcinoma, and Kaposi'ssarcoma.

Other examples include, but are not limited to: tissue viability (e.g.,whether tissue is dead or living, and/or whether it is predicted toremain living); tissue ischemia; malignant cells or tissues (e.g.,delineating malignant from benign tumors, dysplasias, precanceroustissue, metastasis); tissue infection and/or inflammation; and/or thepresence of pathogens (e.g., bacterial or viral counts). Someembodiments include differentiating different types of tissue from eachother, for example, differentiating bone from flesh, skin, and/orvasculature. Some embodiments exclude the characterization ofvasculature.

The levels of certain chemicals in the body, which may or may not benaturally occurring in the body, can also be characterized. For example,chemicals reflective of blood flow, including oxyhemoglobin anddeoxyhemoglobin, myoglobin, and deoxymyoglobin, cytochrome, pH, glucose,calcium, and any compounds that the subject may have ingested, such asillegal drugs, pharmaceutical compounds, or alcohol.

Some embodiments include a distance sensor (not shown) that facilitatespositioning the subject at an appropriate distance from the sensorand/or projector. For example, the system 200 can include a laser rangefinder that provides a visible and/or audible signal such as a lightand/or a beep or alarm, if the distance between the system and thesubject is not suitable for obtaining light from and/or projecting lightonto the subject. Alternately, the laser range finder may provide avisible and/or audible signal if the distance between the system and thesubject is suitable.

An aspect provides a method of diagnosing a medical condition insubject, the subject having a plurality of regions. The methodcomprises: (A) resolving light obtained from each region of skin in theplurality of regions of skin into a corresponding spectrum, therebyconstructing a plurality of spectra. The method further comprises (B)identifying a suspect region of skin in the plurality of regions of skinby obtaining a first indication that a first spectrum in the pluralityof spectra that corresponds to the suspect region includes an indicia ofthemedical condition being present in the suspect region. The methodfurther comprises (C) collecting, after the identifying (B), ameasurement of the suspect region using a contact probe module therebyobtaining a second indication that the medical condition is present inthe suspect region. The contact probe module is configured to obtain asignal from the suspect region. In some embodiments, the secondindication is made based on comparison between the measurement and astored measurement signature corresponding to the medical condition. Insome embodiments the signal is an electrical signal, a magnetic signal,a thermal signal, a mechanical signal, or an optical signal. In someembodiments, the method further comprises (D) displaying an indicatorrepresenting the presence of the medical condition in the suspectregion. In some alternative embodiments, the method further comprises(D) outputting an indicator representing the presence of the medicalcondition in the suspect region to a user, a user interface device, or amonitor; or outputting an indicator representing the presence of themedical condition in the suspect region in user readable form to acomputer readable storage medium or a local or remote computer system;or displaying an indicator representing the presence of the medicalcondition in the suspect region.

Another aspect provides a method of diagnosing a medical condition in asubject, the subject having a plurality of regions. The method comprises(A) identifying a suspect region of skin in the plurality of regions ofskin by obtaining a first indication that a first spectrum in aplurality of spectra that corresponds to the suspect region includes anindicia of a medical condition being present in the suspect region. Theplurality of spectra is constructed by resolving light from each regionof skin in the plurality of regions of skin into a correspondingspectrum. The first indication is made based on a comparison to thefirst spectrum to a stored predetermined spectral signaturecorresponding to the medical condition. The method further comprises (B)collecting, after the identifying (A), a measurement of the suspectregion using a contact probe module thereby obtaining a secondindication that the medical condition is present in the suspect region.The contact probe module is configured to obtain a signal from thesuspect region. The second indication is made based on comparisonbetween the signal and a stored measurement signature corresponding tothe medical condition. In some embodiments, the signal is an electricalsignal, a magnetic signal, a thermal signal, a mechanical signal, or anoptical signal. In some embodiments, the method further comprises (C)displaying an indicator representing the presence of the medicalcondition in the suspect region. In some alternative embodiments, themethod further comprises (C) outputting an indicator representing thepresence of the medical condition in the suspect region to a user, auser interface device, or a monitor; or outputting an indicatorrepresenting the presence of the medical condition in the suspect regionin user readable form to a computer readable storage medium or a localor remote computer system; or displaying an indicator representing thepresence of the medical condition in the suspect region.

Another aspect provides a method of diagnosing a medical condition insubject, the subject having a plurality of regions. the method comprises(A) collecting a measurement of a suspect region using a contact probemodule thereby obtaining an indication that a medical condition ispresent in the suspect region, where the suspect region is identifiedfrom the plurality of regions of skin by obtaining a first indicationthat a first spectrum in a plurality of spectra that corresponds to thesuspect region includes an indicia of the medical condition beingpresent in the suspect region. The plurality of spectra is constructedby resolving light from each region of skin in the plurality of regionsof skin into a corresponding spectrum. The first indication is madebased on a comparison to the first spectrum to a stored predeterminedspectral signature corresponding to the medical condition. The contactprobe module is configured to obtain a signal from the suspect region.The second indication is made based on comparison between themeasurement and a stored measurement signature corresponding to themedical condition. In some embodiments, the signal is an electricalsignal, a magnetic signal, a thermal signal, a mechanical signal, or anoptical signal. In some embodiments, the method further comprisesdisplaying an indicator representing the presence of the medicalcondition in the suspect region. In some alternative embodiments, themethod further comprises outputting an indicator representing thepresence of the medical condition in the suspect region to a user, auser interface device, or a monitor; or outputting an indicatorrepresenting the presence of the medical condition in the suspect regionin user readable form to a computer readable storage medium or a localor remote computer system; or displaying an indicator representing thepresence of the medical condition in the suspect region.

Another aspect provides a method of diagnosing a medical condition in asubject, the method comprising (A) obtaining, at a first distance fromthe subject, light from each region of a first plurality of regions ofthe subject. The method further comprises (B) resolving the lightobtained from each region of the first plurality of regions into acorresponding spectrum, thereby constructing a first plurality ofspectra. The method further comprises (C) determining, based on aspectral characteristic present in a subset of the first plurality ofregions, a second distance from the subject allowing for closerexamination of the subset. The method further comprises (D) obtaining,at a second distance from the subject, light from each region of asecond plurality of regions of the subject, the second plurality ofregions including the subset. The method further comprises (E) resolvingthe light obtained from each region of the second plurality of regionsinto a corresponding spectrum, thereby constructing a second pluralityof spectra. The method further comprises (F) identifying a suspectregion of skin in the plurality of regions of skin by obtaining a firstindication and a second indication. In some instances the firstindication represents that a first spectrum in the first plurality ofspectra that corresponds to the suspect region includes an indicia ofthe medical condition being present in the suspect region. In someinstance the first indication is made based on a comparison to the firstspectrum to a stored predetermined spectral signature corresponding tothe medical condition. In some instances, the second indicationrepresents that a second spectrum in the second plurality of spectrathat corresponds to the suspect region includes an indicia of themedical condition being present in the suspect region. In some instance,the second indication is made based on a comparison to the secondspectrum to a stored predetermined spectral signature corresponding tothe medical condition. The method further comprises (G) collecting,after the identifying (F), a measurement of the suspect region using acontact probe module thereby obtaining a third indication that themedical condition is present in the suspect region. In some instances,the contact probe module is configured to obtain a signal from thesuspect region. In some instances, the third indication is made based oncomparison between the measurement and a stored measurement signaturecorresponding to the medical condition. In some instances, the signal isan electrical signal, a magnetic signal, a thermal signal, a mechanicalsignal, or an optical signal. In some embodiments, the method furthercomprises (H) displaying an indicator representing the presence of themedical condition in the suspect region. In some alternativeembodiments, the method further comprises (H) outputting an indicatorrepresenting the presence of the medical condition in the suspect regionto a user, a user interface device, or a monitor; or outputting anindicator representing the presence of the medical condition in thesuspect region in user readable form to a computer readable storagemedium or a local or remote computer system; or displaying an indicatorrepresenting the presence of the medical condition in the suspectregion.

Another aspect provides a method of characterizing a medical conditionin a subject, the subject having a plurality of regions. The methodcomprises (A) resolving, at a first time, light obtained from eachregion of the plurality of regions into a corresponding spectrum,thereby creating a first plurality of spectra. The method furthercomprises (B) storing the spectra corresponding to the first time. Themethod further comprises (C) resolving, at a second time subsequent tothe first time, light obtained from each region of the plurality ofregions into a corresponding spectrum, thereby creating a secondplurality of spectra. The method further comprises (D) identifying asuspect region among the plurality of regions based on a comparison ofthe second plurality of spectra corresponding to the second time to thefirst plurality of spectra corresponding to the first time. The methodfurther comprises (E) collecting, after the identifying (D), ameasurement of the suspect region using a contact probe module therebyobtaining a third indication that the medical condition is present inthe suspect region. In some embodiments the contact probe module isconfigured to obtain a signal from the suspect region. In someembodiments the third indication is made based on comparison between themeasurement and a stored measurement signature corresponding to themedical condition. In some embodiments the signal is an electricalsignal, a magnetic signal, a thermal signal, a mechanical signal, or anoptical signal. In some embodiments, the method further comprises (F)displaying an indicator representing the presence of the medicalcondition in the suspect region. In some alternative embodiments, themethod further comprises (F) outputting an indicator representing thepresence of the medical condition in the suspect region to a user, auser interface device, or a monitor; or outputting an indicatorrepresenting the presence of the medical condition in the suspect regionin user readable form to a computer readable storage medium or a localor remote computer system; or displaying an indicator representing thepresence of the medical condition in the suspect region.

The illumination subsystem 210, sensor subsystem 230, processorsubsystem 250, and projection subsystem 270 can be co-located (e.g., allenclosed in a common housing). Alternatively, a first subset of thesubsystems can be co-located, while a second subset of the subsystemsare located separately from the first subset, but in operablecommunication with the first subset. For example, the illumination,sensing, and projection subsystems 210, 230, 270 can be co-locatedwithin a common housing, and the processing subsystem 250 locatedseparately from that housing and in operable communication with theillumination, sensing, and projection subsystems. Or, each of thesubsystems can be located separately from the other subsystems. Notealso that storage 240 and storage 252 can be regions of the same deviceor two separate devices, and that processor 238 of the sensor subsystemmay perform some or all of the functions of the spectral analyzer 254and/or the image constructor 256 of the processor subsystem 250.

Note also that although illumination subsystem 210 is illustrated asirradiating an area 201 that is of identical size to the area from whichsensor subsystem 230 obtains light and upon which projection subsystem270 projects the image, the areas need not be of identical size. Forexample, illumination subsystem 210 can irradiate an area that issubstantially larger than the region from which sensor subsystem 230obtains light and/or upon which projection subsystem 270 projects theimage. Also, the light from projection subsystem 270 may irradiate alarger area than sensor subsystem 230 senses, for example in order toprovide an additional area in which the subsystem 270 projects notationsand/or legends that facilitate the inspection of the projected image.Alternately, the light from projection subsystem 270 may irradiate asmaller area than sensor subsystem 230 senses.

Although illumination subsystem 210, sensor subsystem 230, andprojection subsystem 270 are illustrated as being laterally offset fromone another, resulting in the subject being irradiated with light comingfrom a different direction than the direction from which the sensorsubsystem 230 obtains light, and a different direction than thedirection from which the projection subsystem 270 projects the imageonto the subject. As will be apparent to those skilled in the art, thesystem can be arranged in a variety of different manners that will allowthe light to/from some or all of the components to be collinear, e.g.,through the use of dichroic mirrors, polarizers, and/or beamsplitters.Or, multiple functionalities can be performed by a single device. Forexample, the projection subsystem 270 could also be used as theirradiation subsystem 210, with timers used in order to irradiate thesubject and project the image onto the subject at slightly offset times.

In some embodiments, the spectral analyzer 254 has access to spectralinformation (e.g., characteristic wavelength bands and/or normalizedreflectances R_(N)(k)) associated with a wide variety of medicalconditions, physiological characteristics, and/or chemicals. Thisinformation can be stored, for example, in storage 252, or can beaccessed via the Internet (interface not shown). In some embodiments,the spectral analyzer has access to spectral information for a narrowsubset of medical conditions, physiological features, or chemicals, thatis, the system 200 is constructed to address only a particular kind ofcondition, feature, or chemical.

The present invention can be implemented as a computer program productthat includes a computer program mechanism embedded in acomputer-readable storage medium. Further, any of the methods disclosedherein can be implemented in one or more computers or other forms ofapparatus. Examples of apparatus include but are not limited to thedevices depicted in FIGS. 2 and 6. Further still, any of the methodsdisclosed herein can be implemented in one or more computer programproducts. Some embodiments disclosed herein provide a computer programproduct that encodes any or all of the methods disclosed herein. Suchmethods can be stored on a CD-ROM, DVD, magnetic disk storage product,or any other computer-readable data or program storage product. Suchmethods can also be embedded in permanent storage, such as ROM, one ormore programmable chips, or one or more application specific integratedcircuits (ASICs). Such permanent storage can be localized in a server,802.11 access point, 802.11 wireless bridge/station, repeater, router,mobile phone, or other electronic devices.

Some embodiments provide a computer program product that contains any orall of the program modules shown in the Figures. These program modulescan be stored on a CD-ROM, DVD, magnetic disk storage product, or anyother computer-readable data or program storage product. The programmodules can also be embedded in permanent storage, such as ROM, one ormore programmable chips, or one or more application specific integratedcircuits (ASICs). Such permanent storage can be localized in a server,802.11 access point, 802.11 wireless bridge/station, repeater, router,mobile phone, or other electronic devices.

Additional Embodiments

The following includes description of additional optional embodiments ofcomponents of the hyperspectral apparatus and of methods for datacollection and data analysis. It will be understood to one of ordinaryskill in the art that the additional embodiments described therein areapplicable to both the hyperspectral image devices without one or morecontact probes and to the hyperspectral image devices with one or morecontact probes, unless otherwise specified.

In some embodiments, the hyperspectral apparatus has at least oneoperating parameter that is a sensor control, an exposure setting, aframe rate, or an integration rate. In some embodiments, the lightsource of the in the hyperspectral apparatus is controlled by thecontrol software module. In some embodiments, the hyperspectralapparatus is portable and further comprises one or more batteries forpowering the hyperspectral sensor, the control computer and the lightsource. In some embodiments, the hyperspectral apparatus furthercomprises a scan mirror to provide simulated motion for a hyperspectralscan of the skin of the subject.

In some embodiments, the hyperspectral sensor is mounted on a tripod. Insome embodiments, the tripod is a fixed sensor tripod or a fixed sensortripod on wheels. In some embodiments, the hyperspectral sensor ismounted on a mobile rack.

In some embodiments, the hyperspectral apparatus further comprises atrained data analysis algorithm, which is stored in the computerreadable memory, for identifying a region of the subject's skin ofbiological interest using an image obtained by the apparatus.

In some embodiments, the hyperspectral apparatus further comprises atrained data analysis algorithm, which is stored in the computerreadable memory, for characterizing a region of the subject's skin ofbiological interest using an image obtained by the apparatus.

In some embodiments, the hyperspectral apparatus further comprises atrained data analysis algorithm, which stored in the computer readablememory, for determining a portion of a hyperspectral data cube thatcontains information about a biological insult in the subject's skin.

In the preceding embodiments, the trained data analysis algorithm is atrained neural network, a trained support vector machine, a decisiontree, or a multiple additive regression tree.

In some embodiments, the hyperspectral apparatus further comprises analternative sensor other than a hyperspectral sensor. The alternativesensor is a digital camera, a LIDAR sensor, or a terahertz sensor.

In some embodiments, the hyperspectral apparatus further comprises afusion module, stored in the computer readable memory, for fusing animage of a portion of the skin of the subject from the other sensor andan image of a portion of the skin of the subject from the hyperspectralsensor. In some embodiments, the fusion module comprises instructionsfor color coding or greyscaling data from the image of a portion of theskin of the subject from the hyperspectral sensor onto the image of aportion of the skin of the subject from the other sensor. In someembodiments, the fusion module comprises instructions for color codingor greyscaling data from the image of a portion of the skin of thesubject from the other sensor onto the image of a portion of the skin ofthe subject from the hyperspectral sensor. In some embodiments, thefusion module comprises instructions for color coding or greyscalingdata from the image of a portion of the skin of the subject from theother sensor as well as color coding or greyscaling data from the imageof a portion of the skin of the subject from the hyperspectral sensor.

In some embodiments, the hyperspectral apparatus further comprises anintegrated display for displaying data from the hyperspectral sensor anda value of the at least one operating parameter that is controlled bythe control computer. In some embodiments, the integrated displayfurther displays the probabilistic presence of a biological insult tothe skin of the subject.

In some embodiments, the hyperspectral apparatus further comprises aspectral analyzer module, which is stored in the computer readablemedia. The spectral analyzer module comprises instructions fordetermining a boundary of an image of a biological insult in thehyperspectral image. In some embodiments, the boundary of the image ofthe biological insult is manually determined by a user. In someembodiments, the boundary of the image of the biological insult isdetermined by a trained data analysis algorithm.

In some embodiments, the hyperspectral apparatus further comprises acommunications module that comprises instructions for communicating theboundary of the image or the biological insult to a local or remotecomputer over a network connection. In some embodiments, thecommunications module further comprises instructions for communicating aframe of reference of the skin of the subject with the boundary of theimage of the biological insult to the local or remote computer over thenetwork connection.

In one aspect, the present application provides a method of diagnosing amedical condition in a subject, the subject having a plurality ofregions. In some embodiments, the method comprises a) resolving lightobtained from each region of the plurality of regions into acorresponding spectrum; b) obtaining, based on a stored spectralsignature corresponding to the medical condition, a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; c) displaying, if the probability exceeds afirst pre-defined threshold, an indicator representing the probablepresence of the medical condition in the corresponding region; d)accepting user input setting a second pre-defined threshold; and e)displaying, if the probability exceeds the second pre-defined threshold,an indicator representing the probable presence of the medical conditionin the corresponding region.

In one aspect, the present application provides a method of diagnosing amedical condition in a subject, the subject having a plurality ofregions. In some embodiments, the method comprises: a) resolving lightobtained from each region of the plurality of regions into acorresponding spectrum; b) obtaining, based on a stored spectralsignature corresponding to the medical condition, a probability thateach spectrum includes indicia of the medical condition being present inthe corresponding region; c) displaying, if the probability exceeds afirst pre-defined threshold, an indicator representing the probablepresence of the medical condition in the corresponding region; and d)displaying at least one of a type of the medical condition, a categoryof the medical condition, an age of the medical condition, a boundary ofthe medical condition, and a new area of interest for examination.

In one aspect, the present application provides a method of diagnosing amedical condition in a subject, the subject having a plurality ofregions. In some embodiments, the method comprises: a) obtaining, at afirst distance from the subject, light from each region of a firstplurality of regions of the subject; b) resolving the light obtainedfrom each region of the first plurality of regions into a correspondingspectrum; c) determining, based on a spectral characteristic present ina subset of the first plurality of regions, a second distance from thesubject allowing for closer examination of the subset; d) obtaining, ata second distance from the subject, light from each region of a secondplurality of regions of the subject, the second plurality of regionsincluding the subset; e) resolving the light obtained from each regionof the second plurality of regions into a corresponding spectrum; f)obtaining, based on a stored spectral signature corresponding to themedical condition, a probability that each spectrum includes indicia ofthe medical condition being present in the corresponding region; and g)displaying, if the probability exceeds a pre-defined threshold, anindicator representing the probable presence of the medical condition inthe corresponding region.

In one aspect, the present application provides a method of diagnosing amedical condition in a subject, the subject having a plurality ofregions. In some embodiments, the method comprises: a) resolving, at afirst time, light obtained from each region of the plurality of regionsinto a corresponding spectrum; b) storing the spectra corresponding tothe first time; c) resolving, at a second time subsequent to the firsttime, light obtained from each region of the plurality of regions into acorresponding spectrum; d) determining, based on a comparison of thespectra corresponding to the second time to the spectra corresponding tothe first time, that the medical condition had been present at the firsttime although it had not been apparent at the first time; and e)displaying an indicator representing the probable presence of themedical condition in the subject.

In another aspect, the present application provides a hyperspectralapparatus comprising a contact probe module. The contact probe module isin electronic communication with the control computer and is configuredto measure a region of interest of the skin of a subject that isidentified based on an aspect of the hyperspectral image. Further, thecontact probe module comprises a first probe head unit connected to ahandle unit.

In some embodiments, the handle unit in the contact probe module of thehyperspectral apparatus is configured to receive a first probe head unitand a second probe head unit. The first probe head unit is a differenttype of probe head than the second probe head unit. In some embodiments,the contact probe module is configured to obtain more than one type ofsignal from the region of interest of the skin of the subject. In someembodiments, the contact probe module is configured to measure at leasta first type of signal and a second type of signal. The first type ofsignal is different than the second type of signal.

In some embodiments, the contact probe module is configured to measure aregion of interest of the skin of a subject at a plurality of differenttimes.

In one aspect, the present application provides a method of diagnosing amedical condition of a subject using a contact probe module, where thesubject has a plurality of regions of skin of interest. According to themethod, light is first obtained from each region of skin in theplurality of regions of skin of interest and then resolved into acorresponding spectrum using a hyperspectral module, therebyconstructing a plurality of spectra. Further, a suspect region of skinis identified in the plurality of regions of skin by obtaining a firstindication that a first spectrum in the plurality of spectra thatcorresponds to the suspect region includes an indicia of a medicalcondition being present in the suspect region. After the identification,a measurement of the suspect region is collected using the contact probemodule thereby obtaining a second indication that the medical conditionis present in the suspect region. The contact probe module is configuredto obtain a signal from the suspect region. An indicator representingthe presence of the medical condition in the suspect region is thenoutputted to a user, a user interface device, or a monitor.Alternatively, an indicator representing the presence of the medicalcondition in the suspect region in user readable form is outputted to acomputer readable storage medium or a local or remote computer system;or displaying an indicator representing the presence of the medicalcondition in the suspect region.

In some embodiments, the first indication is made based on a comparisonbetween the first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition. The stored predeterminedspectral signature is obtained from a region that is known to have themedical condition.

In some embodiments, the first indication is made based on a comparisonbetween the first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition and the indication is made whenthe first spectrum shares a similarity to the stored predeterminedspectral signature.

In some embodiments, the similarity is determined based on a similaritymetric computed between the first spectrum and the stored predeterminedspectral signature.

In some embodiments, the first indication is made based on a comparisonbetween the first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition and the second indication is madewhen the signal from the suspect region shares a similarity to thestored measurement signature. In some embodiments, the similarity isdetermined based on a similarity metric computed between the signal fromthe suspect region and the stored measurement signature.

In some embodiments, the first indication is made based on a comparisonbetween the first spectrum and a stored predetermined spectral signaturecorresponding to the medical condition.

In some embodiments, the second indication is made based on comparisonbetween the measurement and a stored measurement signature correspondingto the medical condition.

REFERENCES

All references cited herein are hereby incorporated by reference hereinin their entirety and for all purposes to the same extent as if eachindividual publication or patent or patent application was specificallyand individually indicated to be incorporated by reference in itsentirety for all purposes.

Many modifications and variations of this application can be madewithout departing from its spirit and scope, as will be apparent tothose skilled in the art. The specific embodiments described herein areoffered by way of example only, and the application is to be limitedonly by the terms of the appended claims, along with the full scope ofequivalents to which the claims are entitled.

What is claimed:
 1. An apparatus for analyzing the skin of a subject,the apparatus comprising: a hyperspectral sensor for obtaining ahyperspectral image of said subject over a spectrum of wavelengths,wherein the spectrum of wavelengths comprises a first wavelength and asecond wavelength different than the first wavelength; a controlcomputer for controlling the hyperspectral sensor, wherein the controlcomputer is in electronic communication with the hyperspectral sensorand wherein the control computer controls at least one operatingparameter of the hyperspectral sensor, and wherein the control computercomprises a processor unit and a computer readable memory; a controlsoftware module, stored in the computer readable memory and executed bythe processor unit, the control software comprising instructions forcontrolling said at least one operating parameter of the hyperspectralsensor; a spectral calibrator module, stored in the computer readablememory and executed by the processor unit, the spectral calibratormodule comprising instructions for applying a wave-length dependentspectral calibration standard constructed for the hyperspectral sensorto the hyperspectral image collected by the hyperspectral sensor,wherein the applying of a wave-length dependent spectral calibrationstandard constructed for the hyperspectral sensor to the hyperspectralimage collected by the hyperspectral sensor comprises (i) obtaining asensitivity of the hyperspectral sensor responsive to each wavelength inthe spectrum of wavelengths, and (ii) calibrating the hyperspectralimage using each sensitivity of the hyperspectral sensor responsive toeach wavelength in the spectrum of wavelengths, wherein a firstsensitivity of the hyperspectral sensor responsive to the firstwavelength is different than a second sensitivity of the hyperspectralsensor responsive to the second wavelength; and a light source thatilluminates the skin of the subject for the hyperspectral sensor.
 2. Theapparatus of claim 1, wherein the light source comprises a polarizer. 3.The apparatus of claim 2, wherein the hyperspectral sensor comprises across polarizer.
 4. The apparatus of claim 1, wherein the hyperspectralsensor comprises a sensor head, and wherein the control software modulecomprises instructions for moving the sensor head through a range ofdistances relative to the subject, including a first distance thatpermits a wide field view of a portion of the subject's skin, and asecond distance that permits a detailed view of a portion of thesubject's skin.
 5. The apparatus of claim 1, the apparatus furthercomprising: a spectral analyzer module stored in the computer readablememory, the spectral analyzer module comprising instructions forcomparing a spectrum acquired using the hyperspectral sensor to asignature in a plurality of spectral signatures corresponding tocharacterized human lesions.
 6. The apparatus of claim 1, the apparatusfurther comprising: a storage module, stored in the computer readablemedia, wherein the storage module comprises a plurality of spectra ofthe subject's skin taken at different time points; and a spectralanalyzer module, stored in the computer readable media, wherein thespectral analyzer module comprises instructions for using the pluralityof spectra to form a normalization baseline of the skin.
 7. Theapparatus of claim 6, wherein the different time points span one or morecontiguous years up to the life time of the subject.
 8. The apparatus ofclaim 6, wherein the spectral analyzer module further comprisesinstructions for analyzing the plurality of spectra to determine a timewhen a biological insult originated.
 9. The apparatus of claim 8,wherein the biological insult is a lesion.
 10. The apparatus of claim 1,the apparatus further comprising a sensor other than a hyperspectralsensor.
 11. The apparatus of claim 10, wherein the other sensor is adigital camera, a LIDAR sensor, or a terahertz sensor.
 12. The apparatusof claim 1, further comprising: a contact probe module in electroniccommunication with the control computer, wherein the contact probemodule is configured to measure a region of interest of the skin of saidsubject that is identified based on an aspect of the hyperspectralimage, and wherein said contact probe module comprises a first probehead unit connected to a handle unit.
 13. The apparatus of claim 12,wherein said contact probe module further comprises a connector unitconnected to the handle unit.
 14. The apparatus of claim 12, whereinsaid first probe head unit is configured to obtain a signal from aregion of interest of the skin of the subject, wherein said signal is anelectrical, a magnetic, a thermal, a mechanical, a radiometric, aphotometric, or an optical signal.
 15. The apparatus of claim 12,wherein said first probe head unit is an electrical probe, a magneticprobe, a thermal probe, a mechanical probe, a radiometric probe, aphotometric probe or an optical probe.
 16. The method of claim 1,wherein the wave-length dependent spectral calibration standardconstructed for the hyperspectral sensor is obtained by comparingmeasured reflection of light from a standard that is measured by thehyperspectral sensor to expected reflection of light from the standard.